The premise is refreshingly unromantic. Code passing its tests is not the same thing as code being good, and only the first of those has a cheap reward signal. Generate, run hidden tests, score — that is why models write correct slop. Sloppiness needs judgment, so it stays unmeasured, and in projects adding millions of lines a month the humans quietly lose the ability to keep up.

The author, a physicist turned eval engineer, went looking for how the industry measures it and came back disappointed. “With the exception of a few insightful research papers, I was disappointed at how vibes based the industry seems at the moment.”

The approaches that fail, and why:

  • LLM as judge by score. Asking a model to rate code 1–10 is “basically equivalent to a random number generator.”
  • LLM as judge by preference. Show it solutions A and B and it will change its answer when you rename them.
  • Human judges. The best option for readability, but it does not scale to training runs or cross-provider benchmarks.
  • Raw LOC delta. Surprisingly effective — and self-destructing, because optimizing for it makes it meaningless. Goodhart’s law, applied to the author’s own preferred metric.

What survives comes from the SlopCodeBench paper, chosen because it separates legacy code from agent code cleanly:

  • Verbosity: the fraction of lines that are duplicated or flagged as unnecessary, using handcrafted AST-Grep rules. |flagged ∪ clone lines| / LOC
  • Erosion: how much of the codebase mass sits in large, complex functions. Mass is cyclomatic complexity times the square root of source lines; erosion is the mass of functions with complexity above 10 over total mass.

The gap is not subtle. Across established repositories, verbosity averages 0.15 ± 0.06 and erosion 0.31 ± 0.17. For code generated during the SlopCodeBench evaluation, verbosity is 0.33 ± 0.10 and erosion 0.68 ± 0.20 — roughly double on both axes. The author checked his own vibe-coded projects and found verbosity up to 0.4 and erosion up to 0.75, so it is not a benchmark artifact.

The more important finding is why agents do not clean this up themselves, and it is a benchmark-design result rather than a capability one. SlopCodeBench hands out instructions and tests in multiple rounds and erases the model context between checkpoints, which mimics how people actually run agents. Bad decisions then accumulate. On the strict solve rate — every test passing at every checkpoint — state-of-the-art models score 0%.

The conclusion is not that metrics solved evaluation. It is that taste is still in the loop: the heuristics that define verbosity were written by a person, because deciding what counts as unnecessary is the hard part, not detecting it. Which is the right thing to be honest about when your job is measuring a property that resists measurement.

For anyone running agents against a growing codebase, the practical takeaway is two numbers you can trend in CI. Not as a quality gate — as an early warning that the code is taking on the shape of generated code. And a direct counterexample to the assumption that having agents review agents keeps the mess contained.