Runback
← Blog

Why replay-from-step-N is harder than it looks

Runback Team··engineering, architecture

Here's the failure mode that makes replay dangerous if you get the capture wrong: you reconstruct the prompt from your own logs, replay it, watch the bug disappear, ship the fix — and the bug is still there in production. Your reconstruction quietly dropped a tool definition, or got a message role wrong, and you just replayed your best guess at the request, not the request. The fix worked against a request that never actually happened.

1 boundary
where the fully-assembled request exists as a single object — after every merge and transform, before the provider sees it

One boundary sees everything

Runback's SDK doesn't log at the call sites in your code. It wraps the model itself, at the Vercel AI SDK's middleware boundary — wrapGenerate's doGeneratehook. Tool definitions have already been merged in by the SDK. System messages have already been concatenated from wherever they came from. Message arrays have already been transformed into the provider's API format. That's the one place, after all of it, where the request exists as a single complete object, right before it's handed to the provider.

Your code, the wrapLanguageModel capture boundary, and the provideryour agentgenerateText(...)wrapGeneratedoGenerate boundaryrequest fullyassembled hereproviderOpenAI · Anthropic · ...captured → replayable exactly as sent
Merges and transforms your framework does internally already happened by the time doGenerate fires — this is the one point where the request exists as a single, complete object before the provider ever sees it.
agent.ts
const dbg = withDebugger(model, {
  runName: "my-agent",
  input: task,
});

const result = await generateText({
  model: dbg.model,
  tools: dbg.tools(myTools),
  prompt: task,
});

That's the whole integration. withDebugger() wraps the model with middleware that captures the request at that boundary and the response, usage, and latency on the way back — before the request reaches the provider and after the response comes back, with nothing reconstructed in between.

Because the captured request is the literal object the provider received, replaying it means re-issuing exactly that request — not a reconstruction that might have quietly dropped a tool definition or gotten a message role wrong.

So what?

When you edit a message and hit replay, you're changing one field of the actual historical request, not rebuilding an approximation of it from scratch and hoping it matches. The fix you see working in the Replay tab is the fix that actually works in production, because it ran against the same request, not a lookalike. That's also what keeps the capture layer thin enough to be three lines of integration instead of a rewrite of your agent.