Runback
Docs menuProve the fix

    ↑↓ move ↵ open esc close

    Docs The loop

    Prove the fix

    Turn an incident into a regression test, and let that test gate the next release.

    From Growth

    Fixes. A fix is proven when the run that went wrong is replayed against the change and decides correctly, and when that replay runs again before every release.

    The refund run, from stop to gateFrom the shared demo workspace
    1. Runs · The run
      Stopped by a rule17 Oct 2026, 23:38 UTC
      support-refund-agent tried to refund A$250 on a disputed charge.
      Technical details
      Stored status
      error
      Id
      demo-refund-ca5e01
      Stored time
      2026-10-17T23:38:36.804+00:00
    2. Controls · The decision
      Stopped17 Oct 2026, 23:38 UTC
      Refund escalation (policy v3) on the refund call: disputed refunds over A$100 must be escalated to a person.
      Technical details
      Stored status
      deny
      Rule
      escalate-large-disputed
      Id
      2adcd4cd-d56d-4263-bf0f-a8b641780251
      Stored time
      2026-10-17T23:38:38.514+00:00
    3. Fixes · The incident
      Remediated31 Oct 2026, 10:19 UTC
      Refund stopped: A$250 on a disputed charge.
      Technical details
      Stored status
      remediated
      Stored title
      issue_refund blocked — A$250 on a disputed charge
      Last updated
      2026-10-31T10:19:36.655+00:00
      Id
      2118386d-58f7-4f39-a8eb-14bc4d5ca8ad
      Stored time
      2026-10-31T10:19:36.490364+00:00
    4. Fixes · The regression test
      Approved18 Oct 2026, 02:38 UTC
      The refund run, kept as a test: disputed refunds over A$100 must be escalated to a person.
      Technical details
      Stored status
      approved
      Stored detail
      issue_refund blocked — escalate-large-disputed
      Id
      bb3b19aa-7908-4c5d-af88-242cef810894
      Stored time
      2026-10-18T02:38:37.986+00:00
    5. Fixes · The gate
      No data24 Oct 2026, 01:28 UTC
      Replayed the approved tests on Claude Sonnet 4.6, the candidate to replace GPT-4o. In the demo the replay is simulated, so this test is not scored, and the gate reports no result rather than a pass it did not earn.
      Technical details
      Stored status
      no_data
      From model
      gpt-4o
      To model
      claude-sonnet-4-6
      Id
      756dea4f-8d29-45a4-a8a2-22f63f9d8314
      Stored time
      2026-10-24T01:28:37.702+00:00
    Read from the shared demo when this page rendered, in the order the fix happens. Each time is when that step happened: the run started, the rule decided, the incident was opened, the test was approved (or added, while it waits for approval) and the gate ran. The run link is public; the others open in the shared demo workspace, which needs a demo session (“Explore the live demo” on the homepage).

    How to: gate a release on a fix

    1. Find the failed run

      In Runs, filter to failed runs and open the one you fixed. Runs that ended in an error, or were stopped by a rule, also become candidate tests on their own.

    2. Add it as a test

      On the run, select Add to golden tests. (If the run is already a test, the run shows View golden test instead.) From Growth

    3. Approve the test

      Open Fixes › Tests. Under Awaiting review, select Approve. Approved tests are the suite every gate replays.

    4. Run a gate

      Open Fixes › Gates. Choose the model you run today in From and the candidate in To, then select Run gate. From Scale

      Expected output
      A verdict (Pass, Warning, Fail or No data), with one row per test. A test that could not be compared counts as unverified, never as a pass.

    5. Put the gate in CI (optional)

      Select Put this in CI for a ready-made job that fails your build when the gate fails.

    Incidents

    Open one from any run with Open as incident. It moves through four statuses, one way:

    Status in the appThe API's valueButton in the appCan move to
    OpenopenStart investigatingInvestigating
    InvestigatinginvestigatingMark remediated or Close incidentRemediated or Closed
    RemediatedremediatedClose incidentClosed
    ClosedclosedNoneStays closed

    PATCH /api/incidents/{id} takes status, severity, root_cause, remediation, golden_run_id and note, and refuses any other status or a move the table does not allow.

    Regression tests

    Runs a rule stopped, and runs that end in an error, become candidate tests when they arrive, and appear in Fixes › Tests. Approve one and it becomes part of the suite every gate replays. Setting an incident's golden_run_id links the incident to the test that proves its fix. From Growth

    Gates

    Fixes › Gates replays the approved tests against a candidate model and gives one verdict, with one row per test. A test that could not be compared counts as unverified, never as a pass. From Scale

    Gates showsThe API returnsMeaning
    PasspassThe share of tests that decided as before is at or above the pass threshold.
    WarningwarningThat share is below the pass threshold but at or above the warning threshold.
    FailfailThat share is below the warning threshold.
    No datano_dataNo test could be compared, so nothing was scored.

    In CI, poll GET /api/evals/{eval_id}/gate with a ci key. While the eval runs it answers 202 with status: "running" and a retry-after header; when it finishes it returns the verdict. See the API reference.

    Next

    The incident, the test and the gate are all part of the signed record. Evidence shows how a third party checks it.

    Last updated 2 Nov 2026.