An engineer at Earendil set out to answer the question the coding-agent industry mostly hand-waves: how do you actually measure whether AI-written code is bad?

His starting point is that correctness is close to solved, and for a structural reason. Code is easy to check — run it against hidden tests and you get a clean pass/fail signal — so it’s the perfect thing to train a model on. Sloppiness is the opposite: duplicated logic, pointless abstractions, structurally bad decisions that pass every test. Judging that requires human taste, which makes it a genuinely hard measurement problem.

  • The industry default, AI-as-judge, mostly doesn’t work. Asking a model to rate code 1-10 is “basically equivalent to a random number generator,” and asking which of two solutions is better produces preference flips when you simply rename them.
  • Human reviewers give the best-quality signal — the least scalable one. You can’t staff millions of lines of review just to rank model providers.
  • His most effective simple metric turned out to be the change in lines of code, with the honest caveat that optimizing for it would destroy its meaning.
  • Two metrics from the SlopCodeBench paper separate human code from agent code well: verbosity (how much code is duplicated or needlessly wordy) and erosion (how much of the codebase’s weight sits in a few tangled, high-complexity functions).
  • The gap is roughly 2×: established repos score 0.15 verbosity and 0.31 erosion on average, agent-written code 0.33 and 0.68.

The most important finding is about accumulation, not any single generation. SlopCodeBench feeds the agent multiple rounds of instructions and tests with its context wiped between checkpoints, which is how agents are actually used. Bad structural decisions compound across rounds — and on the strict measure, where every test must pass at every checkpoint, even the best models score 0%.

That kills the common reassurance that agents will clean up their own slop. Detecting slop is precisely the part of the loop that still needs a person. When a project adds hundreds of thousands of lines a month, nobody can keep up, and the result is a slow transfer of control from the humans to the machines.