Runback
← Blog

We held a real data race to one fixed schedule

Runback Team··engineering, determinism

A bug that won't reproduce the second time you run it is the most infuriating property of debugging anything concurrent — including AI agents. So before we trusted our replay engine with a single customer's incident, we tried to break its determinism claim on the textbook example of a bug that isn't supposed to reproduce at all: a genuine, lock-free data race.

600, 515, 527
uncontrolled — a different answer every run, out of 600 possible (CI, 6 Oct 2026)
N, N, N
under our scheduler — the same value every run; the value itself awaits the first green CI run after the scheduler fix below

The setup

A racy increment loop across multiple threads, compiled at -O0 so the increment stays a genuine load/add/store read-modify-write — no compiler optimization papering over the race. Run uncontrolled, its final counter value is nondeterministic by construction: threads interleave differently every time, so some increments get lost.

Two threads racing to increment the same counterboth threads hold counter=5thread Aread 5+1 = 6write 6thread Bread 5+1 = 6write 6counter = 6(should be 7)
Both threads read the counter before either writes back. Thread A's increment overwrites thread B's — one increment vanishes. Neither thread did anything wrong; the race is in the gap between read and write.

We ran it three times under Runback's instruction-granularity scheduler — the same class of technique rr and Hermit use to serialize threads onto a single deterministic timeline — and required two things: the result had to be identical across all three runs, and it had to show lost updates (final count below the theoretical max), proving a genuine race actually occurred rather than an accidental full serialization.

The result

linux/race_test.sh — the shape of its report (illustrative: N is not yet recorded)
plain : counter=600 / 515 / 527  (of 600)   ← real, nondeterministic data race
sched : counter=N / N / N        (of 600)   ← must be identical every run, and N<600 = lost updates

PASS identical result across 3 runs under the scheduler
PASS the result shows lost updates (N < 600) — a genuine data race occurred

What that guarantees is narrower than “replay”, and worth saying exactly. Under Runback's scheduler the same binary gives the same result on every run, because the scheduler imposes one fixed schedule. It does not record the schedule of an uncontrolled run and replay it: a counter=515 seen outside the scheduler cannot be reproduced. And it covers single-process pthread programs with no blocking waits between threads.

This runs on every push. If it ever regresses — if run two doesn't match run one — the build goes red, the same way a broken unit test would. And it did: until a fix in October 2026, this tier was intermittently non-deterministic under CPU load, because the scheduler admitted new threads whenever the kernel happened to report them. Its own CI found it.

That's not a simulation — it's a real data race held to one fixed schedule, and it runs in our CI (linux/race_test.sh, part of the determinism-proof workflow). It's one tier in a deeper research stack: libc interposition, ptrace-level syscall interception, a deterministic thread scheduler, and instruction-precise preemption via the PMU retired-conditional-branch counter, each tested independently and layered to reach this result. All of these tiers, libc interposition included, are research programs in native/ and linux/, not something Runback runs on your agents: the product's replay is the cassette captured by the SDK.

So what?

This isn't the product — it's the stress test. But it's the same determinism engine behind Runback's replay-from-any-step feature. When you replay a captured LLM step from a real agent run, you're trusting that everything around that step held still — that the only thing different is the edit you made. What this test shows is that the scheduler can hold a racy program to one schedule, run after run; it is not evidence that an arbitrary concurrent program's past run can be replayed.

We'd rather show that in public CI than assert it in a pitch deck — the proof run is published at github.com/letsRunback/runback-proofs, and you can check any sealed record yourself at runback.dev/verify.

The public record shows the instability that fix addresses: 28 of the last 35 daily runs passed; the 7 failures are listed beside the passes (as of 3 Nov 2026). On those runs the same binary produced different counter values across the three runs that are meant to be identical. The CI history is where to watch it hold.