The claim

Dan Luu dismantles the increasingly-cited claim that dynamic languages are more token-efficient for coding agents.

The source: an Alderson post claiming a 2.6x gap between C and Clojure, with J at ~70 tokens. It’s now so entrenched that Google’s AI summary repeats it verbatim.

The method

He pre-registered his guesses before running evals: 95% confidence the dynamic-vs-static claim won’t hold, 98% the “weird language” J-supremacy won’t hold. Then he tested with real work:

  • A complete zstd decoder from the RFC — agents in a container, no internet, holdout tests
  • A modified Pandoc ProgramBench with holdout scoring

The results

  • At medium effort, dynamic languages look better; at ultra effort, results go mixed with more static languages among the best
  • The extreme ratios from toy problems don’t survive contact with real work
  • Mainstream languages show a weak-to-moderate correlation with correctness and cheapness — labs concentrate synthetic RL data on popular languages, so “use an obscure dense language” is bad advice for normal LLM users

The broken benchmark

The sharpest part is the appendix: the widely-cited mame/ai-coding-lang-bench is broken.

Tests executed a non-existent path, and the first Go run symlinked its executable there — so every later language’s scoring actually ran Go’s binary. Rescored correctly, Rust’s “difficult language” failures become a perfect score.

The takeaway

You can’t draw strong claims about particular languages from a few evals. Most circulating language-LLM lore is wrong or unsupported.