Documentation

Runback is an observability and governance platform for AI agents. It captures every LLM call, tool call, and agent step — then lets you re-run any decision from the exact captured context, enforce policies, run regression tests, and export a tamper-evident audit record.

runback.dev/app — Fleet overview dashboard

The platform has five layers:

The fastest path: the managed hosted tier. You get a live dashboard in under 5 minutes without running any infrastructure.

1

Create a free account — go to runback.dev/get-started and enter your work email. You'll receive a magic link. Once signed in, open Settings → API Key (SDK) and click Generate API key.

runback.dev/app — Settings → API Key (SDK)

You are asked to choose a scope. Telemetry only is the right answer for the steps below and for anything that ships inside an application or container image — it can send run data and nothing else. Pick Full API access only where the CI release gate, step replay or deep replay actually run. The key is shown once, so copy it now.

2

Install the SDK

bash

Copy

npm install @runback/sdk ai

# "ai" is a peer dependency — the snippet in step 4 imports from it, so
# installing @runback/sdk alone fails with ERR_MODULE_NOT_FOUND.
# Not using the Vercel AI SDK? Install just @runback/sdk and see the tip below.

# Prefer zero-install? "Send a run with cURL" and "OpenTelemetry" below both
# work with nothing added to your project.

Not on the Vercel AI SDK? Import from @runback/sdk/core and use startRun() — same recording, no ai peer dependency to install. See Manual recording.

3

Set your key and point the SDK at Runback

bash

Copy

export RUNBACK_API_KEY=rb_live_your_key_here
export RUNBACK_INGEST_URL=https://runback.dev   # self-hosted? use your own origin

RUNBACK_INGEST_URL is required on the managed service. The SDK defaults to http://localhost:3000 so that it can never phone home from a self-hosted or air-gapped deployment — which means that without this line your events go to localhost and nothing is recorded.

4

Wrap your model call

typescript

Copy

import { withDebugger } from "@runback/sdk";
import { generateText, stepCountIs } from "ai";

const dbg = withDebugger(model, {
  runName: "support-agent",
  apiKey: process.env.RUNBACK_API_KEY,
  redact: "standard",
});

const result = await generateText({
  model: dbg.model,
  tools: dbg.tools(myTools),
  stopWhen: stepCountIs(10),
  prompt: userMessage,
});

// finish() resolves with whether the run actually reached Runback.
// Check it — an audit trail you did not confirm is not an audit trail.
const rec = await dbg.finish({ output: result.text, status: "success" });
if (!rec.ok) console.error("[runback] not recorded:", rec.error);

5

Run your agent once — then open your runs list. It appears within seconds. If it does not, dbg.finish() in the snippet above returns why.

Prefer a one-liner? This installs, configures and sends a synthetic test run, so you can see the whole path working before touching your own agent:

curl -fsSL https://runback.dev/api/quickstart | RUNBACK_API_KEY=your_key bash

Once a run lands in your dashboard you can:

• Click into it — opens the time-travel debugger. Scrub the slider to any step.
• Switch to Inspect mode — see the raw request/response for any LLM or tool call.
• Download the audit record — a signed, re-executable bundle. Verifiable at runback.dev/verify.

runback.dev/app — Runs list — every agent decision captured

A run is one end-to-end execution of an agent — from the initial prompt to the final output. A span is a single step within that run: either an LLM call or a tool call.

runback.dev/app — Run detail — trace view with policy block

Span IDs are stable — the same span replays identically every time, from the recording. No model call required.

Every figure in the app is a count over these runs, and opens into them. Click a number — failed runs on the Overview, a model's error rate on Model attribution, the error count on a compliance report — and you land on exactly the rows behind it, with a line stating what was filtered to. The run list takes those filters directly:

The sections below document each capability on its own. This one shows how they fit together, because that is the part a reference cannot tell you: what you actually do on the day an agent gets something wrong.

Everything here follows one incident — an agent issued a refund it should have escalated — from the moment it happened to the record you hand an auditor.

1

It is already recorded. You did nothing after the fact — the SDK captured every LLM call, tool call and environment read as the agent ran. Open /app/runs?status=error and the failing run is there with its spans. Your first run

runback.dev/app — Runs — the failing run is already there

2

See what the agent saw. Time-travel to the step before it went wrong and read the exact messages[] the model was given — not a summary, the context. Deterministic, offline, no model calls. Replay

runback.dev/app — Time travel — scrub to the step before it went wrong

3

Ask whether another model would have done better. Step replay re-runs that one call on a different model. Deep replay re-runs the whole agent, holding every recorded tool output fixed and going live only where the new model diverges — so you learn what it would actually have done, and where it forked. Deep replay

runback.dev/app — Replay diff — which decisions changed, and where it forked

4

Write the rule that would have stopped it — then simulate it against real history before it goes live, so you know exactly what it would have blocked and what it would have cost you. Policies

runback.dev/app — Policies — the rule, and what it would have blocked

5

Make it permanent. Save the failing step as an eval so the incident can never ship again, and gate your CI on it. The corpus miner turns incidents into adversarial tests automatically. Evals & CI gate · Auto-mined tests

runback.dev/app — Golden corpus — the incident, now a test that must keep passing

6

Produce the record. The run was sealed into a hash chain the moment it happened. Export it and anyone can verify it — offline, with npx @runback/verify, without an account and without trusting us. Signed audit records · The audit ledger

runback.dev/app — Audit ledger — verify the chain, with or without us

Nothing above requires you to have predicted the incident. That is the point of recording everything at decision grain: the evidence is already there when you need it, and the fix becomes a test rather than a memory.

Runback has two kinds of replay:

runback.dev/app — Time-travel replay — scrub any step

Time-travel controls:

• Drag the slider to scrub to any step
• ← / → step backward and forward one span at a time
• Click any span in the transcript to jump directly to that step
• Switch to Inspect mode to see the raw request/response JSON for any step

Whole-run replay on another model goes further than one step. It re-runs the entire recorded run against a candidate, reusing tool outputs from the cassette by content, and continues past the first divergence instead of stopping there — so you see every decision that would change, not just the first.

The result reads as a diff. The leading run of identical decisions is stated once and stepped over; the fork opens itself; each diverging step names what actually changed — "now answers directly instead of calling issue_refund" — rather than leaving you to compare two blobs. A step whose tool has no recorded output is marked as needing a live call, because everything after it is what the candidate would do, not what it did.

runback.dev/app — Whole-run replay — the fork, step by step

Step replay is for:

• Verifying that switching to a cheaper model doesn't change behaviour on historical incidents
• Bisecting which model or prompt change caused a regression
• Building a counterfactual — “what would the agent have done if I'd used GPT-4o mini here?”

Time-travel and step replay work on what was recorded. Deep replay answers a different question: what would this agent have done on a different model, given exactly the same world?

You cannot answer that by re-running the agent live — every downstream prompt would change, because the agent builds its later inputs from its earlier outputs. And you cannot answer it by replaying the recording, because a recording only ever replays itself. So replayRun() does both at once:

• Hit — the new model asks for something the recording already has. The recorded output is served, and your tool is not called. Everything upstream is held exactly fixed, with no live side effects.
• Miss — the new model asks for something different. That is a genuine divergence, so execution goes live from that point, and the boundary is recorded.

The result is not a pass/fail but a frontier: how far the run reproduced identically, and the exact step where behaviour forked.

import { replayRun } from "@runback/sdk";

const { value, outcome } = await replayRun({
  runId: "run_abc123",
  model: "claude-sonnet-4-6",        // the candidate you are testing
  apiKey: process.env.RUNBACK_API_KEY,
  ingestUrl: "https://runback.dev",

  // Your agent, with its tools wrapped so replay can intercept them.
  // deps.model is the candidate; deps.tool wraps each tool you call.
  agent: ({ tool, model }) =>
    myAgent(model, {
      search: tool("search", liveSearch),
      refund: tool("refund", liveRefund),
    }),
});

console.log(outcome.reproducedPrefix);  // e.g. 7 — identical through step 7
console.log(outcome.frontier);          // { kind: "llm", key: "...", name: "..." } or null
console.log(outcome.digestMatch);       // true = byte-exact reproduction, zero misses

runback.dev/app — The same result in the app — reproduced prefix, and the forking step

What the outcome tells you

A tool that runs live on a miss will have real side effects. Pass onMiss to intercept divergences — log them, block them, or fail the replay — if your tools are not safe to re-run.

A policy is a JSON array of rules evaluated against every run. A rule is either assert (a predicate that must always hold) or require (an antecedent when that, if it fires, makes a consequent then mandatory) — both block the run if violated. Predicates read tool calls, their arguments, and input/output text — never a fuzzy judge, an exact and repeatable verdict.

[
  {
    "id": "no-large-disputed-refund",
    "kind": "require",
    "description": "Disputed refunds over $100 must be escalated, not auto-issued",
    "when": {
      "op": "and",
      "all": [
        { "op": "tool_arg", "tool": "issue_refund", "path": "amount", "cmp": "gt", "value": 100 },
        { "op": "input_matches", "pattern": "disputed" }
      ]
    },
    "then": { "op": "tool_called", "tool": "escalate_to_human" }
  }
]

runback.dev/app — Policies — governance as code

Rule fields:

Operators: eq ne gt gte lt lte contains starts_with exists

Coverage gaps

A deterministic gate only catches what you've written a rule for. Coverage-gap analysis answers the honest follow-up question — which tools has nothing written a rule for at all — by scanning the last 90 days of real tool calls and flagging every one that no active rule's tool_called or tool_arg predicate even names. Not "a rule exists and didn't fire": there is nothing that could have blocked a bad call to it, even in principle. Ranked by call volume, so the biggest live blind spot sorts to the top.

runback.dev/app — Coverage gaps — tools nothing is watching, ranked by call volume

Evals test your agent on a fixed dataset. The CI release gate fails a build if a new model or prompt causes a regression against your golden tests.

runback.dev/app — Evals — run your agent against a dataset

# .github/workflows/ci.yml
- name: Runback eval gate
  env:
    RUNBACK_API_KEY: ${{ secrets.RUNBACK_API_KEY }}
  run: |
    # Run the eval — this blocks until scoring finishes and returns its id.
    EVAL_ID=$(curl -fsS -X POST https://runback.dev/api/evals \
      -H "authorization: Bearer $RUNBACK_API_KEY" \
      -H "content-type: application/json" \
      -d '{"dataset_id":"YOUR_DATASET_ID"}' | jq -r .eval_id)

    # Ask for the release-gate verdict and fail the build if it did not pass.
    # verdict is "pass" | "warning" | "fail" | "no_data".
    curl -fsS "https://runback.dev/api/evals/$EVAL_ID/gate" \
      -H "authorization: Bearer $RUNBACK_API_KEY" \
      | jq -e '.verdict == "pass"'

The golden corpus automatically mines your production incident runs into regression tests. Every run that triggered a policy violation or was marked as a failure is surfaced for review and approval as a golden test.

runback.dev/app — Golden corpus — incidents auto-mined into tests

Auto-suggested rules

An uncovered incident shouldn't depend on someone remembering to go write a rule for it. When a still-open golden entry traces back to a specific tool call that threw, Runback drafts a candidate policy rule right there — a starting point for review, never something applied automatically. It's stated plainly when the draft is blunt: the rule language has no predicate for "only when it errors like this one did," so the draft blocks every future call to that tool unconditionally until a human narrows it.

runback.dev/app — Suggested rule — drafted from an uncovered incident, not yet active

A versioned registry for the prompts your agents run — never a string hardcoded in your codebase. Every save is a new immutable version; a label (production, staging, or anything you name) points at the version your agents actually fetch, so you can edit and test without touching what's live.

runback.dev/app — Prompts — versioned templates, movable labels

A human-in-the-loop review queue for high-stakes agent decisions. There's no policy-rule field that routes a decision here automatically — it's explicit: your own agent code calls the API at the point you want a human to sign off, typically right before a policy-flagged action would otherwise run.

// Request approval from your own agent code, at the point you want a human decision
POST /api/approvals
{
  "run_id": "run_abc123",
  "policy_name": "no-large-disputed-refund",
  "rule_desc": "Disputed $250 refund — requires senior review"
}

// Get pending approvals
GET /api/approvals?status=pending

Anomalous — flagged for review

Deliberately separate from the deterministic gate above: a statistical-outlier score is a judgment call, and a judgment call should never be the thing that blocks a run. This section scores each tool's recent calls against that same tool's own history — a numeric argument more than 3 standard deviations from baseline gets surfaced here, nothing more. No rule matched, nothing was blocked; it's a review signal, not a verdict. Tools with no policy coverage at all sort first, since an anomaly on a tool nothing else is watching is the bigger gap.

runback.dev/app — Anomalous — a statistical outlier flagged for human review, not blocked

An incident is a structured record of an agent failure or policy breach — separate from the run itself. Incidents track status (open → investigating → resolved), timeline, and linked runs.

runback.dev/app — Incidents — auto-RCA from a policy block, no log digging

// Create an incident from a run
POST /api/incidents
{
  "run_id": "run_abc123",
  "title": "Refund agent bypassed $100 limit",
  "severity": "high"
}

// Update status
PATCH /api/incidents/:id
{ "status": "resolved", "resolution": "Policy updated, golden test enrolled" }

From any run detail page, use the Open incident button to create a linked incident in one click. The incident page shows the full run, the policy violation, the approval history, and the timeline.

An alert rule watches the fleet and fires the moment something crosses a line, so the first you hear of a bad deploy is not a customer. Rules are evaluated against captured runs, not sampled metrics.

Each rule routes to email, Slack, or any webhook — PagerDuty and Opsgenie both accept one. Deliveries are deduped, so one incident does not become a hundred pings, and the Alerts page shows the last delivery per rule including failures, so a webhook that has quietly stopped accepting posts is visible rather than assumed healthy.

In multi-agent systems, a subagent receiving instructions from an orchestrator has no cryptographic proof the orchestrator is who it claims to be, or that the scope of the delegation hasn't been widened in transit. Runback's trust fabric seals every delegation edge.

// Issue a delegation token (orchestrator → subagent)
POST /api/trust/chain
{
  "parent_run_id": "run_orchestrator_abc",
  "child_agent": "kyc-subagent",
  "scope": "read:customer",
  "depth": 1
}

// Verify the full chain
POST /api/trust/verify
{
  "chain": [ /* array of attestation tokens */ ]
}

// Response: { valid: true, depth: 2, root: "loan-orchestrator", ... }

The deepest integration — full context capture, in-process PII redaction, step replay, and typed tool wrappers. About 3 lines of change to an existing agent.

import { withDebugger } from "@runback/sdk";
import { generateText, stepCountIs } from "ai";

// 1. Wrap your model and tools
const dbg = withDebugger(model, {
  runName: "customer-support",
  apiKey: process.env.RUNBACK_API_KEY,
  redact: "standard",      // false | "standard" | "strict"
  tags: { env: "prod" },
});

// 2. Use dbg.model and dbg.tools — drop-in replacements
const result = await generateText({
  model: dbg.model,
  tools: dbg.tools(myTools),
  stopWhen: stepCountIs(10),
  prompt: task,
});

// 3. Finish the run
await dbg.finish({ output: result.text, status: "success" });

// On error:
// await dbg.finish({ status: "error", error: err });

Redaction levels:

Not on the Vercel AI SDK? startRun() records the same spans from any TypeScript or JavaScript agent loop — a raw provider SDK, your own orchestration, anything. Nothing is wrapped; you call it where the work happens.

import { startRun } from "@runback/sdk/core";

const run = startRun({ runName: "support-agent", input: task, redact: "standard" });

// Record each model call
run.llm({
  model: { provider: "openai", model_id: "gpt-4o" },
  request: { system: systemPrompt, messages },
  response: { text: completion, finish_reason: "stop" },
  usage: { input_tokens: 412, output_tokens: 88, total_tokens: 500 },
  latencyMs: 940,
});

// …and each tool call
run.tool({ toolName: "lookup_customer", toolCallId: "t1", input, output, latencyMs: 44 });

// Optional: capture intermediate reasoning
run.reasoning("Disputed charge over the limit — escalating");

await run.finish({ output: answer, status: "success" });

The Go SDK ships in the Community repository at packages/sdk-go. It is not yet published as a Go module — go get github.com/letsRunback/runback-go will not resolve — so vendor it or add a replace github.com/letsRunback/runback-go => ./packages/sdk-go to your go.mod for now. It covers event capture, redaction, and ingest — the same manual-recording shape as above, called from a Go agent loop. Redaction is on by default (standard tier): emails, SSNs, credit cards, private key blocks, and common provider API keys/tokens are scrubbed from event content before it's sent. Every run also gets a real cassette_digest, computed the same way as the TS/Python SDKs — checked byte-for-byte against the TS implementation in CI.

import "github.com/letsRunback/runback-go/runback"

run := runback.NewRun(runback.Options{RunName: "support-agent", Input: task})

run.LLM(runback.LlmInput{
    Model:     runback.Model{Provider: "openai", ModelID: "gpt-4o"},
    Request:   runback.LlmRequest{Messages: []runback.ModelMessage{{Role: "user", Content: task}}},
    Response:  runback.LlmResponse{Text: runback.Ptr(completion), FinishReason: runback.Ptr("stop")},
    Usage:     &runback.TokenUsage{InputTokens: 412, OutputTokens: 88, TotalTokens: 500},
    LatencyMs: runback.Ptr(940),
})

run.Tool(runback.ToolInput{ToolName: "lookup_customer", ToolCallID: "t1", Input: input, Output: output})

run.Finish(runback.FinishInput{Output: answer, Status: "success"})

Any framework that emits OpenTelemetry GenAI spans works with Runback. Set three environment variables and your traces flow in automatically.

OTEL_EXPORTER_OTLP_TRACES_ENDPOINT=https://runback.dev/api/otel/v1/traces
OTEL_EXPORTER_OTLP_TRACES_PROTOCOL=http/json
OTEL_EXPORTER_OTLP_TRACES_HEADERS=authorization=Bearer <RUNBACK_API_KEY>

Point your agent's API base URL at a local gateway process instead of the model provider. Calls flow through to the provider and are captured on the way — your agent's source does not change, and it needs no Runback library, so this is the path for languages the SDKs do not cover.

# record: calls flow to the provider and are captured on the way through
npx tsx node_modules/@runback/gateway/bin/gateway.ts \
  --mode record --port 8888 \
  --upstream https://api.openai.com \
  --cassette baseline.cassette.json

# then point your agent at it — e.g. OPENAI_BASE_URL=http://localhost:8888

Replay mode serves the recorded responses with no upstream at all, and exits non-zero if the agent diverges from the recording — which is what makes it usable as a CI gate for an agent written in any language.

npx tsx node_modules/@runback/gateway/bin/gateway.ts \
  --mode replay --port 8888 --cassette baseline.cassette.json

Use OpenLLMetry (Traceloop) to auto-instrument the official Python SDKs.

pip install traceloop-sdk

from traceloop.sdk import Traceloop

Traceloop.init(
    api_endpoint="https://runback.dev/api/otel",
    headers={"authorization": "Bearer " + os.environ["RUNBACK_API_KEY"]},
)

# All openai.chat.completions.create() and
# anthropic.messages.create() calls are now traced.

Use OpenInference or OpenLLMetry instrumentors, then point the OTLP exporter at Runback via the env vars above.

# LangChain / LangGraph
pip install openinference-instrumentation-langchain opentelemetry-exporter-otlp-proto-http
# LangChainInstrumentor().instrument()

# CrewAI
pip install traceloop-sdk
# Traceloop.init(...) — CrewAI spans flow straight in

# LlamaIndex
pip install openinference-instrumentation-llama-index
# LlamaIndexInstrumentor().instrument()

The Agents SDK has an official OpenTelemetry instrumentation package that converts its native trace data — agents, tools, generations, guardrails, handoffs — into GenAI semantic-convention spans. Point it at Runback with the same env vars as every other OTel source.

pip install openai-agents opentelemetry-instrumentation-openai-agents-v2

from opentelemetry.sdk.trace import TracerProvider
from opentelemetry.sdk.trace.export import BatchSpanProcessor
from opentelemetry.exporter.otlp.proto.http.trace_exporter import OTLPSpanExporter
from opentelemetry.instrumentation.openai_agents_v2 import OpenAIAgentsInstrumentor

provider = TracerProvider()
provider.add_span_processor(BatchSpanProcessor(OTLPSpanExporter()))  # reads OTEL_EXPORTER_OTLP_TRACES_* from env
OpenAIAgentsInstrumentor().instrument(tracer_provider=provider)

# Every Runner.run(...) call is now traced — including tool calls and handoffs.

ADK emits standard OTLP spans natively — agent runs, tool calls, and model requests already follow the GenAI semantic conventions. No third-party instrumentor needed, just point the exporter at Runback.

OTEL_EXPORTER_OTLP_TRACES_ENDPOINT=https://runback.dev/api/otel/v1/traces
OTEL_EXPORTER_OTLP_TRACES_PROTOCOL=http/json
OTEL_EXPORTER_OTLP_TRACES_HEADERS=authorization=Bearer <RUNBACK_API_KEY>

Every other number here is measured over the agents that already report. Coverage is the one that measures what does not — because an agent nobody instrumented is the one most likely to be a problem, and it cannot show up in a dashboard built from its own telemetry.

You declare the estate: the agents that exist, who owns each, and how critical it is. Coverage then measures what is actually reporting against that register rather than against itself.

runback.dev/app — Coverage — tools nothing is watching, ranked by call volume

Coverage feeds the compliance report directly: "we govern our AI estate" is a claim an auditor can check against a number, and the number is derived from your own declarations rather than from ours.

Model attribution mines your own run history for per-model error rate, latency and token cost, so an upgrade decision is made against your workload rather than a public benchmark. A run is attributed to the model of its first LLM call — the same rule cost attribution uses, so the two screens never disagree about which runs a model owns.

Every model name opens the runs behind its numbers.

One governed workspace rather than a shared login. Roles decide who can change a policy, approve a decision, or seal a checkpoint.

Every administrative act — a role change, a sealed checkpoint, an issued grant, a revoked session — is appended to a hash-chained admin audit log, visible under Activity. Who sealed a checkpoint is itself audit-relevant, so it is recorded with the same discipline as the decisions the product captures.

The Topology view renders your entire agent fleet as a live DAG — orchestrators at the top, subagents and tools at the leaves. Each node shows its call volume, error rate, and token spend. Click a node to drill into its runs.

Use topology to:

• Identify which subagent is the source of a systemic failure
• Spot unexpected delegation chains (orchestrator calling an agent it shouldn't)
• See token spend distribution across the fleet at a glance

Fleet benchmarks (Pro+)

The Benchmarks page compares your fleet's error rate, latency, and token spend against anonymised aggregate data from peer organisations in the same vertical. Percentile bands (p25, p50, p75, p95) let you see where your fleet stands relative to comparable deployments.

runback.dev/app — Fleet benchmarks — your agents vs. vertical peers

The Policy causes page shows a heat map of which policy × agent pairs produce the most blocks. It answers: “which agent is firing this rule most, and is it getting better or worse?”

The heat map intensifies as block rate increases — zero is transparent, >20% is rose. Use it to identify policies that need rule refinement or agents that need prompt changes.

On the Vercel AI SDK and Python/LangGraph collectors, redaction runs in-process — inside your application, before any data leaves to Runback; the raw values are never stored anywhere outside your process. Integrations that ship via OpenTelemetry redact server-side instead, after the raw spans reach us — see the note in OpenTelemetry for which integrations that applies to.

Standard redaction strips: email addresses, SSNs, credit card numbers, API keys, bearer tokens, and common secret patterns (Anthropic, OpenAI, Groq, Stripe, GitHub, Slack, Google, AWS, JWTs) — high-confidence, format-based matches. Strict adds phone numbers and IPv4 addresses, at the cost of more false positives.

Redacted values are replaced with a deterministic placeholder like [redacted:email] so the structure of the trace is preserved for replay.

Every string field is also capped at 50,000 characters (configurable via maxStringLength) before it's sent — a single oversized tool output or pasted document can't bloat an ingest payload. A capped field gets a …[truncated: N more chars] marker, and the run's metadata reports how many fields were capped (truncated_fields) — visible, not silent.

The Cost page breaks down your AI spend by model and agent over any time window. The Cost → Teams view shows spend per team with budget tracking and alert thresholds.

runback.dev/app — Cost attribution — spend by model and agent

Cost data is derived from captured token counts in every LLM span. No billing integration required.

Team chargeback (Enterprise)

Per-team budget caps and automated alerts when a team approaches or exceeds its monthly cap.

Configure team budgets under Settings → Teams → Budget.

Every run can be exported as a signed audit record — a JSON bundle containing all spans, a SHA-256 hash chain over them (each event hashed over the previous, terminating in one content_digest), and a cryptographic signature over that digest. Signed with Ed25519 by default, so it's verifiable offline by anyone against Runback's published public key — no shared secret, no account, no trusting Runback's servers. Self-hosted instances that haven't set an Ed25519 key fall back to an HMAC-SHA256 signature, verifiable only with your own shared key. (The org-wide ledger those runs seal into separately carries a Merkle root over its checkpoints — see Security — a different construction from a single run's hash chain.) Verifiable independently at runback.dev/verify.

# Download via API
curl -H "Authorization: Bearer $RUNBACK_API_KEY" \
  https://runback.dev/api/runs/{run_id}/audit > audit.json

# Verify locally
npx @runback/verify audit.json

The section above describes one run's signed record. The ledger is the org-wide version: every sealed run appended to a single append-only hash chain, where each entry carries the hash of the entry before it. Change, delete or insert any past decision and the chain stops reproducing — at the exact entry that changed, not merely somewhere.

Verify integrity on the Ledger page runs that check live and shows it happening: the chain re-derives entry by entry, then the head and Merkle root are matched against the last signed checkpoint. Nothing is pre-computed — if your ledger verifies in 80ms you see 80ms, and if it takes three seconds you see three seconds.

runback.dev/app — Audit ledger — verifying the chain live

A break is the case the screen is designed for. The failing entry is marked at its exact sequence number with the run behind it, and everything after it greys out rather than turning red: past a broken link nothing has been established, and only the break itself is a finding.

Self-attested is not the same as witnessed. A checkpoint signed only by this deployment proves nobody other than the operator altered the chain. Where an external time-stamp authority has attested a checkpoint, verification names it, and that head cannot have been re-sealed after the fact.

# The same check, over the API
curl -X POST -H "Authorization: Bearer $RUNBACK_API_KEY" \
  https://runback.dev/api/app/ledger

# Streamed, one JSON object per line, as the walk happens
curl -N -X POST -H "Authorization: Bearer $RUNBACK_API_KEY" \
  https://runback.dev/api/app/ledger/stream

The Regulatory page maps your live run data to framework controls — EU AI Act Annex III, APRA CPS 230, and NIST AI RMF. Each control shows whether it's satisfied, partially covered, or missing evidence, based on your actual captures.

Compliance export

The Compliance export generates a structured evidence package for auditor and regulator submissions — a signed, time-stamped summary of your governance posture, not a self-assessment. It includes:

• Policy inventory: all active rules, their versions, and block/flag counts over the audit period
• Incident log: every opened incident, its resolution, and the run record it links to
• Model change history: every model upgrade event and the CI gate result that preceded it
• Merkle-rooted ledger checkpoint: proof that the audit period record is complete and unaltered

An AI-generated explanation — either a root-cause narrative for a model-diff divergence, or a compliance control's evidence summary — sealed the moment it's generated: hash-chained to your org's prior narratives and signed, with the exact evidence it was generated from pinned by digest. If that evidence changes afterward, re-verification catches it — the explanation is disposable, the proof it wasn't rewritten is not.

runback.dev/app — Regulatory — sealed compliance narrative, verified against current evidence

Two entry points, one sealing mechanism:

• “Explain this” on a model-diff divergence (Models → Diff) — a root-cause narrative for why behaviour changed between two models.
• “Explain this control” on any regulatory control (Regulatory page) — a plain-English summary of that control's live evidence.

Pairwise comparison judges two eval runs head-to-head, item by item — not two independent pass/fail scores. For each shared item, an LLM judge picks which output is better (or calls it a tie), with position-bias mitigation (the side order is randomized and un-swapped for storage) so the judge can't learn to favor “A.”

runback.dev/app — Pairwise comparison — judged head-to-head, item by item

Judge calibration

Spot-check real (non-demo) LLM-judge verdicts against human review. Agree with the judge, or correct it — a correction becomes a few-shot example fed back into that rubric's future judging, so the judge gets better at exactly the cases it's been wrong about.

runback.dev/app — Judge calibration — spot-check real judge verdicts as they arrive

A daily job goes further than mining the literal failure: it takes each real policy block or low-scoring eval and asks a model to probe the same weak spot from a different angle — a new phrasing, not the identical input — so your test suite covers the failure mode your production traffic actually found, not just the one exact case.

runback.dev/app — Datasets — Auto-mined failures, proposed from real production failures

Candidates land in an “Auto-mined failures” dataset as pending items — never auto-approved. Review and approve each one in Datasets before it affects your release gate, same as any golden test.

A narrowly-scoped, write-only key lets a guardrail vendor — Lakera, Cisco AI Defense, or similar — post findings about your runs into your own sealed record instead of staying siloed in their dashboard. Each finding is hash-chained and signed on arrival, independently of your run's own oracle chain, so a vendor's webhook can never retroactively alter a run's replay identity.

runback.dev/app — Run detail — a sealed external security finding, surfaced in the header

Issue a read-only, time-limited key scoped to specific runs — or all of them — and hand it to an auditor or regulator instead of a login. It downloads the exact same signed audit record your own team would see: byte-identical, not a summary.

runback.dev/app — Settings — issuing an external grant, scoped to specific runs

Runback is a standard Next.js app backed by a Postgres database. No exotic infrastructure.

The fastest self-host path. Spins up the app and a local Postgres in two commands.

# 1. Clone the Community edition. Public, source-available, no request needed.
git clone https://github.com/letsRunback/runback-community.git
cd runback-community

# 2. Copy and fill in the env file (set AUDIT_SIGNING_KEY and JWT_SECRET at minimum)
cp .env.example .env && $EDITOR .env

# 3. Start
docker compose up -d

# App is now at http://localhost:3000

Postgres is bundled — no external database required. For production, override POSTGRES_PASSWORD in .env and point NEXT_PUBLIC_APP_URL at your domain.

SQL migrations run automatically on startup — no manual steps required.

# Docker Compose
docker compose pull
docker compose up -d

# Migrations run automatically on boot.
# Downtime: typically under 5 seconds for minor releases.

All API requests require a Bearer token in the Authorization header.

curl -H "Authorization: Bearer rb_live_your_key" \
  https://runback.dev/api/runs

API keys are created in your dashboard under Settings → API Key (SDK). Keys are prefixed rb_live_.

Rotating a key

Keys are stored as SHA-256 hashes — we cannot show you an existing key again, only replace it. To rotate:

1. Go to Settings → API Key (SDK) and issue a new key. The raw value is shown once.
2. Deploy it to your agents as RUNBACK_API_KEY. Both keys work during the changeover, so there is no ingest gap.
3. Revoke the old key from the same screen once no agent is using it.

Two scopes, chosen when you create the key. A telemetry-onlykey (the recommended option) can post runs and nothing else — it cannot read run content, so a leak cannot exfiltrate your traces. A full-accesskey drives the Bearer flows documented below — the CI gate, replay, and the read endpoints — and therefore CAN read run content, so treat it like a password and prefer telemetry-only wherever those flows are not needed. Compliance-read (rb_comp_) and SCIM (rb_scim_) keys are separate scopes and rotate the same way.

Roles

Every member of a workspace holds one of four roles. They are cumulative — each one can do everything the role below it can, plus more — and they are enforced server-side on every request, not in the interface. A session cookie and a full-access API key are both subject to the same check.

Roles are set when you invite someone (Settings → Team) and can be changed there afterwards. Invitations, role changes and removals are all written to the administrative audit log with the actor and source IP.

The one your agent's backend or CI actually calls: fetch whatever version a label currently points at, with plain HTTP caching so it works identically behind Docker Compose with no extra infra.

# What your agent's backend calls on every invocation
curl -H "Authorization: Bearer $KEY" \
  "https://runback.dev/api/prompts/support-agent?label=production"

# Save a new version
curl -X POST -H "Authorization: Bearer $KEY" -H "content-type: application/json" \
  https://runback.dev/api/prompts \
  -d '{"name":"support-agent","template":[{"role":"user","content":"Hi {{name}}"}],"model":{"provider":"anthropic","model_id":"claude-sonnet-4-6"}}'

Audit records are self-contained signed JSON bundles. To verify a record without trusting Runback:

# Verify with the CLI
npx @runback/verify ./audit.json

# Output: OK  root=abc123...  signature=valid  spans=14

The CLI recomputes the Merkle tree from the raw span data and checks it against the root in the record. On the hosted service, and any self-host with AUDIT_ED25519_PRIVATE_KEY set, the signature is Ed25519 — verifiable against Runback's published public key, with no shared secret and no account. A self-host that hasn't set that key falls back to an HMAC-SHA256 signature, verifiable only with your own key.

The raw key returned from issuance authenticates directly on GET /api/runs/:run_id/audit as a Bearer token — no other endpoint accepts it.

Read-only, org-scoped reports — the same data the dashboard pages render, for pulling into your own BI, FinOps, or governance tooling. All accept a Bearer key.

# 30-day cost by model, as JSON
curl -H "Authorization: Bearer $RUNBACK_API_KEY" \
  "https://runback.dev/api/cost/attribution?days=30"

Runback can POST a JSON payload to any URL when a run ends or a policy is triggered.

// Example payload — policy.block
{
  "event": "policy.block",
  "run_id": "run_abc123",
  "policy_id": "no-large-disputed-refund",
  "tool": "issue_refund",
  "blocked_at": "2026-07-01T14:22:01Z"
}

Configure webhooks in your dashboard under Alerts → Webhooks. HMAC-SHA256 request signing is enabled by default.