Runback
← Blog

Simulate a policy against history before you enforce it live

Runback Team··product, policy

The usual way a policy rule reaches production is: someone writes it, someone approves it, it goes live, and the first time anyone finds out whether it's too strict or too loose is when it blocks — or fails to block — a real call. That feedback loop runs on production traffic, which is a bad place to discover your regex was slightly wrong.

100 runs
of real recorded history a candidate rule is checked against before it ever gates a live call

Evaluate against history, not against the future

Runback's policy simulator takes a candidate rule set and replays it — deterministically, with no model calls and no re-execution — against the org's most recent recorded runs. Each run's LLM decisions are evaluated in their original sequence, because a rule that accumulates state across tool calls (a running total, say) has to see the calls in the order they actually happened to give the same verdict enforcement would have given live.

A candidate policy evaluated against 100 recorded runs before it ever gates a live callcandidate rulenot yet enforced100 recorded runsrecorded LLM decisions,evaluated in original orderno model calls, no replaywould-block listnamed runs, not a count
The output names the exact runs a candidate rule would have blocked, not just a block-rate percentage — a risk owner can open each one and judge if that's correct.
policySim.ts
export async function simulatePolicyOverRuns(
  orgId: string, rules: PolicyRule[], limit = 100,
): Promise<PolicySimResult> {
  const runs = await recentRuns(orgId, limit);
  const affected: PolicySimRun[] = [];
  for (const run of runs) {
    const decisions = await recordedDecisions(run.run_id);
    const v = wouldBlock(decisions, rules);
    if (v.blocked) affected.push({ run_id: run.run_id, name: run.name, detail: v.detail });
  }
  return { total: runs.length, blocked: affected.length, affected, /* ... */ };
}

The output isn't a block-rate number — it's the actual list of runs the rule would have caught, each one a real link you can open and inspect. That distinction matters: a block rate tells you a rule is aggressive, but only the named runs let a risk owner check whether it's aggressive correctly — catching the incident it was written for, not an unrelated batch of harmless calls that happen to share a keyword.

A rule simulated clean against 100 real runs and then enforced live isn't a guess that got lucky — it's a rule that already survived the exact traffic it's about to run against, just without the ability to actually block anything yet.

Where this fits

This is the same mechanism behind the "simulate a policy against history, then enforce it live" capability described on the competitive comparison page — most observability tools can show you a trace after the fact, but none of them let you test a not-yet-enforced rule against a real production history before it's live. Read-only tools can't do this because there's nothing to simulate against without a real recorded decision history to evaluate the candidate rule over.