Stephen and Cassandra Chin demonstrate Tracetoon, a tool for preparing evidence-backed pull-request descriptions. Their PR Bench quality and token figures describe a project-internal comparison, not an independent benchmark.
Reconstruct what the agent actually did
- AI-written PR text can claim a test passed even when the run was concurrent with another command or checked an earlier version of the branch.
- Tracetoon collects relevant agent-session traces and code context for the branch. Cassandra’s initial loop still introduced unsupported claims during later rewrites.
- A reviewer needs links to the commands, their timing, actual output and the submitted revision, not a polished prose summary of the diff.
Give the writer a connected evidence brief
- The speakers use an embedded graph database called Relate to connect change rationale, code, commands and checks before drafting.
- They kept a stronger writer for prose and used less expensive reviewers for narrower claim checks. “Zero LLM cost” describes the graph query alone; building and reviewing the PR still uses models.
- They report a modest increase in judge-rated description quality and about half the token use in their own PR Bench comparisons. The talk does not establish independent replication, general savings or that the graph is the sole cause.
“He claims tests passing … actually that’s not what the code did.” — Stephen Chin