Observe, Replay, Gate, Audit — the mechanism that turns a black-box agent into one you can prove.
See it applied to regulated production →Runback keeps a signed record of everything an agent does — so you can reproduce any run, find what broke, and test a fix before release.
Every step records the precise context the model was handed — system prompt, conversation, tools — captured at the model boundary, with PII redacted in-process. A failed run opens on the step that broke. Click any step:
Re-execution holds tools, retrieval, and context fixed and runs the model fresh — a different output is signal, not a recording glitch. When something breaks, bisect the timeline to the exact change in log₂ tries.
Flagged the $85k application for human review — correctly applied the $50k limit.
Approved the same $85k application — ignored the $50k limit entirely.
12 revisions, one regression. Bisect finds it in 4 probes, not 12:
On a real run, each probe compares the model's decision — which tool, with what arguments — not tool or environment behavior. A regression caused by a changed tool response, not the model's choice, won't be found by this search.
The same production failure, resolved with a log pipeline and with Runback. Not a customer story — the mechanism below is exactly what a real incident uses.
Save the step as an eval that must keep passing. Simulate a candidate policy against real history to see what it would have blocked, then enforce it in-process — blocked before it runs. A breach becomes a red row, never an incident.
Export a complete, tamper-evident record of any run — every event in a SHA-256 hash chain, signed. Change one byte and verification fails. The artifact an auditor actually asks for.
SHA256(prev_hash + canonical({kind, key, output}))Runs also seal into an org-wide ledger — one signed root proves any run is in the log, without handing over the rest.
A run that breaches policy or errors auto-enrolls as a golden test — deduped, so each distinct failure is one test that grows your suite by itself.
A daily job goes further: it takes each real failure and asks a model to probe the same weak spot from a different angle. Every proposal lands as pending — never auto-approved — until a human reviews it.
Walk the loan-approval-agent incident — all 5 steps, the policy block, the cassette →
Logs record the outcome. Re-execution shows the reasoning — the exact context, retrieval, and messages[] the model saw. Root cause in 4m 23s, not 3h 48m.
Demo scenario — walk the exact step · walk it →The capture layer takes a sprint. The golden corpus, fleet benchmarks and policy history take years — and can't be imported, back-filled or bought.
Auto-mined adversarial corpus — the actual mechanism · read how it works →EU AI Act Art. 12, APRA CPS 230, NIST AI RMF — one record, built from the first run rather than retrofitted when the auditor asks.
EU AI Act, Article 12 — read the law · the law ↗one Runback cassette · satisfies all three
Behavioral history only builds forward. Every month you delay is data you can never recover.
Fleet benchmarks, model diff, policy library · what's in Pro →Illustrative — based on common production patterns, not a measured customer average. loan-approval-agent · 2:47 AM · every month you delay is behavioral data you can never recover
Your model calls happen inside your own application. Hook it up once; everything else runs on the trace it produces.
A ~3-line SDK wrap or your existing OpenTelemetry traces. A post-hook — your real model call runs untouched.
const dbg = withDebugger(model)
const res = await generateText({
model: dbg.model,
})
await dbg.finish({
status: "success",
})Secrets, keys, emails, card numbers and SSNs are scrubbed before a trace is sent anywhere.
Postgres you control — self-hosted or our managed cloud. Delete a run and it's gone.
Every feature reads the same captured trace. No extra wiring, no second integration.
Replay and evals re-execute a step, so they use a model key configured server-side. Observe, time-travel, and audit need no model key at all.
import { withDebugger } from "@runback/sdk";
import { generateText, stepCountIs } from "ai";
const dbg = withDebugger(model, { runName: "support-agent", redact: "standard" });
const res = await generateText({
model: dbg.model,
tools: dbg.tools(myTools),
stopWhen: stepCountIs(8),
prompt: task,
});
await dbg.finish({ output: res.text, status: "success" });Every run now shows up ready to observe, replay, gate, and audit. See all integrations →
Walk a real failing run step by step — no signup.