Yusuke Tanaka’s celld case study makes deterministic simulation testing concrete: run production event handlers, but let a simulator choose the order of requests, tasks, timers, and storage completions. Explore different executions, then replay a failure with the same code, settings, and random seed.
The interesting failure involved an alarm and the object-storage entry needed to wake its cell:
- Deleting a 10:00 alarm persisted the SQLite change and returned success, leaving wake-entry cleanup pending.
- A new 10:05 alarm reused the still-present 10:00 wake entry, avoiding another storage write.
- The old cleanup finally ran and deleted the entry the new alarm needed. Both client operations had already succeeded.
- A node restart before 10:05 could then prevent the alarm from running.
The fix changed the ownership model: every new alarm setting gets its own wake entry, so delayed cleanup for an older alarm cannot delete the newer one’s entry.
The testing lessons are just as useful:
- Separate event selection from handling so simulated schedules exercise production logic.
- Control storage failures and delays, task scheduling, and time—not merely request inputs.
- Express acknowledged promises as invariants and check them after every action.
- Keep the discovered execution as a regression test. Checking only the final state can miss a guarantee that briefly failed and later recovered.
This is a better debugging story than adding sleeps until a race becomes hard to reproduce. It makes the failing order inspectable and turns the repair into a structural change rather than a timing assumption.
The limits matter: celld’s simulator was still under development and not public when the article appeared. Its coverage depends on the faults it explores and the guarantees its checker actually encodes.