Runback
← Blog

We deterministically reproduced a real data race

Runback Team··engineering, determinism

A bug that won't reproduce the second time you run it is the most infuriating property of debugging anything concurrent — including AI agents. So before we trusted our replay engine with a single customer's incident, we tried to break its determinism claim on the textbook example of a bug that isn't supposed to reproduce at all: a genuine, lock-free data race.

220, 204, 184
uncontrolled — a different answer every run, out of 600 possible
400, 400, 400
under our scheduler — identical, three runs in a row

The setup

A racy increment loop across multiple threads, compiled at -O0 so the increment stays a genuine load/add/store read-modify-write — no compiler optimization papering over the race. Run uncontrolled, its final counter value is nondeterministic by construction: threads interleave differently every time, so some increments get lost.

Two threads racing to increment the same counterboth threads hold counter=5thread Aread 5+1 = 6write 6thread Bread 5+1 = 6write 6counter = 6(should be 7)
Both threads read the counter before either writes back. Thread A's increment overwrites thread B's — one increment vanishes. Neither thread did anything wrong; the race is in the gap between read and write.

We ran it three times under Runback's instruction-granularity scheduler — the same class of technique rr and Hermit use to serialize threads onto a single deterministic timeline — and required two things: the result had to be identical across all three runs, and it had to show lost updates (final count below the theoretical max), proving a genuine race actually occurred rather than an accidental full serialization.

The result

linux/race_test.sh — output
plain : counter=220 / 204 / 184  (of 600)   ← real, nondeterministic data race
sched : counter=400 / 400 / 400  (of 600)   ← identical every run, and 400<600 = lost updates

PASS data race reproduced DETERMINISTICALLY — identical result across 3 runs
PASS the result shows lost updates (400 < 600) — a genuine data race, reproduced exactly

200 of 600 updates — a third of them — vanish into the race, and they vanish the exact same way three times in a row. That's the whole trick: not preventing the race, reproducing it, on demand, byte for byte.

This runs on every push. If it ever regresses — if run two doesn't match run one — the build goes red, the same way a broken unit test would. Nobody has to remember to check.

That's not a simulation and it's not cherry-picked — it's a real data race, reproduced exactly, every time, and it runs in our CI (linux/race_test.sh, part of the determinism-proofworkflow). It's one tier in a deeper stack: libc interposition, ptrace-level syscall interception, a deterministic thread scheduler, and instruction-precise preemption via the PMU retired-conditional-branch counter, each proven independently and layered to reach this result.

So what?

This isn't the product — it's the stress test. But it's the same determinism engine behind Runback's replay-from-any-step feature. When you replay a captured LLM step from a real agent run, you're trusting that everything around that step held still — that the only thing different is the edit you made. If our replay engine can pin down something as slippery as a genuine data race, three for three, that's the confidence you're actually buying when you hit “replay.”

We'd rather prove that on every commit than assert it in a pitch deck — the full, live CI history is published at runback.dev/verify.