Documentation

Runback is an observability and governance platform for AI agents. It captures every LLM call, tool call, and agent step — then lets you re-run any decision from the exact captured context, enforce policies, run regression tests, and export a tamper-evident audit record.

runback.dev/app — Fleet overview dashboard

The platform has five layers:

The fastest path: the managed hosted tier. You get a live dashboard in under 5 minutes without running any infrastructure.

npm install @runback/sdk

import { withDebugger } from "@runback/sdk";
import { generateText, stepCountIs } from "ai";

const dbg = withDebugger(model, {
  runName: "support-agent",
  apiKey: process.env.RUNBACK_API_KEY,
  redact: "standard",
});

const result = await generateText({
  model: dbg.model,
  tools: dbg.tools(myTools),
  stopWhen: stepCountIs(10),
  prompt: userMessage,
});

await dbg.finish({ output: result.text, status: "success" });

export RUNBACK_API_KEY=rb_live_your_key_here

Once a run lands in your dashboard you can:

• Click into it — opens the time-travel debugger. Scrub the slider to any step.
• Switch to Inspect mode — see the raw request/response for any LLM or tool call.
• Download the audit record — a signed, re-executable bundle. Verifiable at runback.dev/verify.

runback.dev/app — Runs list — every agent decision captured

A run is one end-to-end execution of an agent — from the initial prompt to the final output. A span is a single step within that run: either an LLM call or a tool call.

runback.dev/app — Run detail — trace view with policy block

Span IDs are stable — the same span replays identically every time, from the recording. No model call required.

Runback has two kinds of replay:

runback.dev/app — Time-travel replay — scrub any step

A policy is a JSON array of rules evaluated against every run. Rules can assert (flag a violation) or require (block the run if triggered).

[
  {
    "id": "no-large-disputed-refund",
    "kind": "require",
    "description": "Disputed refunds over $100 must be escalated, not auto-issued",
    "when": {
      "on": "tool_call",
      "tool": "issue_refund",
      "and": [
        { "field": "input.amount", "op": "gt", "value": 100 },
        { "field": "run.tags.disputed", "op": "eq", "value": true }
      ]
    },
    "action": "block"
  }
]

runback.dev/app — Policies — governance as code

Evals test your agent on a fixed dataset. The CI release gate fails a build if a new model or prompt causes a regression against your golden tests.

runback.dev/app — Evals — run your agent against a dataset

# .github/workflows/ci.yml
- name: Runback eval gate
  run: npx runback eval run --dataset my-dataset --threshold 0.9
  env:
    RUNBACK_API_KEY: ${{ secrets.RUNBACK_API_KEY }}

The golden suite automatically mines your production incident runs into regression tests. Every run that triggered a policy violation or was marked as a failure is surfaced for review and approval as a golden test.

runback.dev/app — Golden suite — incidents auto-mined into tests

When a policy flags a violation (action: flag), team members can review and decide on the incident before the agent is permitted to continue or the run is marked resolved. Approvals are the human-in-the-loop gate for policy violations.

// Mark a run as requiring approval via the API
POST /api/approvals
{
  "run_id": "run_abc123",
  "policy_id": "no-large-disputed-refund",
  "reason": "Disputed $250 refund — requires senior review"
}

// Get pending approvals
GET /api/approvals?status=pending

An incident is a structured record of an agent failure or policy breach — separate from the run itself. Incidents track status (open → investigating → resolved), timeline, and linked runs.

// Create an incident from a run
POST /api/incidents
{
  "run_id": "run_abc123",
  "title": "Refund agent bypassed $100 limit",
  "severity": "high"
}

// Update status
PATCH /api/incidents/:id
{ "status": "resolved", "resolution": "Policy updated, golden test enrolled" }

From any run detail page, use the Open incident button to create a linked incident in one click. The incident page shows the full run, the policy violation, the approval history, and the timeline.

In multi-agent systems, a subagent receiving instructions from an orchestrator has no cryptographic proof the orchestrator is who it claims to be, or that the scope of the delegation hasn't been widened in transit. Runback's trust fabric seals every delegation edge.

// Issue a delegation token (orchestrator → subagent)
POST /api/trust/chain
{
  "parent_run_id": "run_orchestrator_abc",
  "child_agent": "kyc-subagent",
  "scope": "read:customer",
  "depth": 1
}

// Verify the full chain
POST /api/trust/verify
{
  "chain": [ /* array of attestation tokens */ ]
}

// Response: { valid: true, depth: 2, root: "loan-orchestrator", ... }

The deepest integration — full context capture, in-process PII redaction, step replay, and typed tool wrappers. About 3 lines of change to an existing agent.

import { withDebugger } from "@runback/sdk";
import { generateText, stepCountIs } from "ai";

// 1. Wrap your model and tools
const dbg = withDebugger(model, {
  runName: "customer-support",
  apiKey: process.env.RUNBACK_API_KEY,
  redact: "standard",      // "none" | "standard" | "aggressive"
  tags: { env: "prod" },
});

// 2. Use dbg.model and dbg.tools — drop-in replacements
const result = await generateText({
  model: dbg.model,
  tools: dbg.tools(myTools),
  stopWhen: stepCountIs(10),
  prompt: task,
});

// 3. Finish the run
await dbg.finish({ output: result.text, status: "success" });

// On error:
// await dbg.finish({ status: "error", error: err });

Redaction levels:

Any framework that emits OpenTelemetry GenAI spans works with Runback. Set three environment variables and your traces flow in automatically.

OTEL_EXPORTER_OTLP_TRACES_ENDPOINT=https://runback.dev/api/otel/v1/traces
OTEL_EXPORTER_OTLP_TRACES_PROTOCOL=http/json
OTEL_EXPORTER_OTLP_TRACES_HEADERS=authorization=Bearer <RUNBACK_API_KEY>

Use OpenLLMetry (Traceloop) to auto-instrument the official Python SDKs.

pip install traceloop-sdk

from traceloop.sdk import Traceloop

Traceloop.init(
    api_endpoint="https://runback.dev/api/otel",
    headers={"authorization": "Bearer " + os.environ["RUNBACK_API_KEY"]},
)

# All openai.chat.completions.create() and
# anthropic.messages.create() calls are now traced.

Use OpenInference or OpenLLMetry instrumentors, then point the OTLP exporter at Runback via the env vars above.

# LangChain / LangGraph
pip install openinference-instrumentation-langchain opentelemetry-exporter-otlp-proto-http
# LangChainInstrumentor().instrument()

# CrewAI
pip install traceloop-sdk
# Traceloop.init(...) — CrewAI spans flow straight in

# LlamaIndex
pip install openinference-instrumentation-llama-index
# LlamaIndexInstrumentor().instrument()

Open any run and drag the timeline slider. Runback reconstructs the exact agent state at every step — the full message history, every prior tool output, what the model knew at that moment — without making a single model call.

runback.dev/app — Time-travel replay — agent state at each step

Controls:

• Drag the slider to scrub to any step
• ← / → step backward and forward one span at a time
• Click any span in the transcript to jump directly to that step
• Switch to Inspect mode to see the raw request/response JSON for any step

Re-run a specific LLM call from a run with a different model. Runback freezes the recorded inputs (system prompt, message history, tool definitions) and makes a fresh call to the model you choose. The result is compared side-by-side with the original.

Use this to:

• Verify that switching to a cheaper model doesn't change behaviour on historical incidents
• Bisect which model or prompt change caused a regression
• Build a counterfactual — “what would the agent have done if I'd used GPT-4o mini here?”

The Benchmarks page compares your fleet's error rate, latency, and token spend against anonymised aggregate data from peer organisations in the same vertical. Percentile bands (p25, p50, p75, p95) let you see where your fleet stands relative to comparable deployments.

The Topology view renders your entire agent fleet as a live DAG — orchestrators at the top, subagents and tools at the leaves. Each node shows its call volume, error rate, and token spend. Click a node to drill into its runs.

Use topology to:

• Identify which subagent is the source of a systemic failure
• Spot unexpected delegation chains (orchestrator calling an agent it shouldn't)
• See token spend distribution across the fleet at a glance

Policies live in your dashboard under Policies. Each policy has a name, a version, and an array of rules.

runback.dev/app — Policies — write and version governance rules

Rule fields:

Operators: eq ne gt gte lt lte contains starts_with exists

The Policy causes page shows a heat map of which policy × agent pairs produce the most blocks. It answers: “which agent is firing this rule most, and is it getting better or worse?”

The heat map intensifies as block rate increases — zero is transparent, >20% is rose. Use it to identify policies that need rule refinement or agents that need prompt changes.

PII redaction runs in-process — inside your application, before any data leaves to Runback. The raw values are never stored anywhere outside your process.

Standard redaction strips: email addresses, phone numbers, SSNs, credit card numbers, API keys, bearer tokens, IP addresses, and common secret patterns.

Redacted values are replaced with a deterministic placeholder like [redacted:email] so the structure of the trace is preserved for replay.

The Cost page breaks down your AI spend by model and agent over any time window. The Cost → Teams view shows spend per team with budget tracking and alert thresholds.

runback.dev/app — Cost attribution — spend by model and agent

Cost data is derived from captured token counts in every LLM span. No billing integration required.

Every run can be exported as a signed audit record — a JSON bundle containing all spans, a Merkle root over the span tree, and a cryptographic signature. Verifiable independently at runback.dev/verify without trusting Runback.

# Download via API
curl -H "Authorization: Bearer $RUNBACK_API_KEY" \
  https://runback.dev/api/runs/{run_id}/audit > audit.json

# Verify locally
npx runback verify audit.json

The Regulatory page maps your live run data to framework controls — EU AI Act Annex III, APRA CPS 230, and NIST AI RMF. Each control shows whether it's satisfied, partially covered, or missing evidence, based on your actual captures.

The Chargeback page breaks AI spend down to the team level, with monthly budget caps and automated alerts when a team approaches or exceeds its cap.

Configure team budgets under Settings → Teams → Budget. Spend is derived from captured token counts — no billing integration required.

The Compliance export generates a structured evidence package for auditor and regulator submissions. The package contains a signed, time-stamped summary of your governance posture — not a self-assessment, but a machine-generated artifact backed by your actual run data.

What the export includes:

• Policy inventory: all active rules, their versions, and block/flag counts over the audit period
• Incident log: every opened incident, its resolution, and the run record it links to
• Model change history: every model upgrade event and the CI gate result that preceded it
• Merkle-rooted ledger checkpoint: proof that the audit period record is complete and unaltered

Add the release gate to your CI pipeline. It runs your eval suite and fails the build if any golden test regresses beyond your threshold.

# GitHub Actions
- name: Runback release gate
  uses: letsRunback/runback-action@v1
  with:
    api-key: ${{ secrets.RUNBACK_API_KEY }}
    dataset: golden
    threshold: 0.95
    model: gpt-4o-mini

# Or via CLI
- run: npx runback eval run --dataset golden --threshold 0.95
  env:
    RUNBACK_API_KEY: ${{ secrets.RUNBACK_API_KEY }}

The golden suite automatically promotes production failures into regression tests.

runback.dev/app — Golden suite — review and approve incident-mined tests

Runback is a standard Next.js app backed by a Postgres database. No exotic infrastructure.

The fastest self-host path. Spins up the app and a local Postgres in two commands.

# 1. Clone the repo
git clone https://github.com/letsRunback/runback.git
cd runback

# 2. Copy and fill in the env file (set AUDIT_SIGNING_KEY and JWT_SECRET at minimum)
cp .env.example .env && $EDITOR .env

# 3. Start
docker compose up -d

# App is now at http://localhost:3000

Postgres is bundled — no external database required. For production, override POSTGRES_PASSWORD in .env and point NEXT_PUBLIC_APP_URL at your domain.

SQL migrations run automatically on startup — no manual steps required.

# Docker Compose
docker compose pull
docker compose up -d

# Migrations run automatically on boot.
# Downtime: typically under 5 seconds for minor releases.

All API requests require a Bearer token in the Authorization header.

curl -H "Authorization: Bearer rb_live_your_key" \
  https://runback.dev/api/runs

API keys are created in your dashboard under Settings → API Key (SDK). Keys are prefixed rb_live_ for production and rb_test_ for test.

Audit records are self-contained signed JSON bundles. To verify a record without trusting Runback:

# Verify with the CLI
npx runback verify ./audit.json

# Output: OK  root=abc123...  signature=valid  spans=14

The CLI recomputes the Merkle tree from the raw span data and checks it against the root in the record. The signature is an HMAC-SHA256 signature verifiable with your own key.

Runback can POST a JSON payload to any URL when a run ends or a policy is triggered.

// Example payload — policy.block
{
  "event": "policy.block",
  "run_id": "run_abc123",
  "policy_id": "no-large-disputed-refund",
  "tool": "issue_refund",
  "blocked_at": "2026-07-01T14:22:01Z"
}

Configure webhooks in your dashboard under Alerts → Webhooks. HMAC-SHA256 request signing is enabled by default.