Runback
← Blog

How we auto-mine production incidents into regression tests

Runback Team··engineering, product

The standard failure mode for agent regression suites is that nobody writes them. Unit tests get written against the happy path someone imagined in advance; the failure that actually happens in production — the one your users hit — usually never becomes a test at all. It gets fixed, everyone moves on, and the exact same bug quietly comes back three prompt revisions later because nothing was checking for it.

1 signature
per distinct failure — a thousand identical incidents dedupe to one golden-suite entry

The corpus is mined, not written

When a run finishes with a runtime policy block or an unhandled error, Runback computes a signature over the reason and the failing detail and upserts it into the golden suite — deduped on (org_id, signature), so the thousandth occurrence of the same failure doesn't create a thousandth test. It links back to the "Production incidents" dataset automatically. Nobody has to remember to write the test; the incident writes it.

A production incident becomes a permanent regression test, deduped by signaturerun failspolicy block or errorsignature computeddedup key, not run idgolden entryre-checked on every suite runnext incident with the same signature — no new test, streak continues
A thousand occurrences of the same underlying failure collapse to one test — the corpus grows with genuinely new failure modes, not with noise.
golden.ts
export async function enrollIfBad(
  orgId: string, runId: string, runName: string,
  events: TraceEvent[], digest: string,
): Promise<void> {
  const bad = detectBad(runName, events);
  if (!bad) return;
  await sb.from("ad_golden").upsert(
    { org_id: orgId, run_id: runId, reason: bad.reason,
      signature: bad.signature, detail: bad.detail,
      baseline_digest: digest, status: "active" },
    { onConflict: "org_id,signature", ignoreDuplicates: true },
  );
}

Running the suite has two distinct modes. Integrity mode re-executes each captured incident and checks that it still reproduces its sealed cassette — a pure determinism check, no model calls, catching accidental drift in the replay engine itself. Candidate mode re-runs each incident against a specific model and reports whether the same bad decision recurs or has changed — the direct answer to "if I ship this model upgrade, does the incident I already fixed come back?"

A streak of good results is a claim that survived N policy revisions, not just N clock ticks — the suite tags every run with a snapshot of which policy version was live at the time, so "this has passed for three weeks" means something specific.

Why dedup on signature, not run

Deduping on the run itself would mean the corpus size tracks incident volume, not incident variety — a noisy week produces a bloated suite that's expensive to run and mostly redundant. Deduping on signature means the suite grows exactly as fast as your agents discover genuinely new ways to fail, which is the only growth rate that makes a regression suite worth maintaining instead of worth ignoring.