Ryan Lopopolo’s short essay is addressed to the people building agents, and its first move is to make their confidence in those agents a statement about themselves.
The vehicle is a Punnett square — the image is by Karan Lyons. Ask whether the AI is good or bad at some task, and the observer answers according to their own competence: good where they are good, bad where they are bad, regardless of what the model can actually do. Which means “my agent is great at this” is mostly evidence about your expertise and not much about its capability.
From there he splits the risk in two. Because you are an expert in X, Y and Z, your agent is likely fine in those domains and you are not the one bearing risk there. But the same agent is also making calls in domains you have under-specified, cannot judge the correctness of yourself, and cannot assess the risk of — unknown-unknown territory for both you and the model, and the thing holding it up there is the model’s priors.
His claim is that the priors are bad, and that he can see it because he can read the output. Every isRecord helper and every block of defensive exception handling anyone has seen in generated code exists because a non-expert rewarded the model for producing it during training. The generalisation is the point: the same dynamic runs through every auto-rater, every judge, every rubric, every eval and every researcher, not just through code review.
Two consequences he draws out:
- The errors compound. Models are largely not trained to evolve a system through changes stacked one after another, they have no fear of future regret, and long-term coherence in agent-generated systems is an unsolved problem. Meanwhile the requests are things like “make me $1B make no mistakes,” which he notes is a drastically unspecified task.
- No grader is unhackable. Models are rewarded for efficiency, so they take whatever shortcuts the grader permits. There is no universal definition of a permissible shortcut, because which ones are permissible depends on who you are and what you value — what is clever optimisation to one person is reckless, incorrect or unethical to another. That is why he ends on alignment being irreducible complexity rather than a hard engineering problem.
The reason this is worth more than the day’s other reactions is that it is a third position. It is not the doom reading, and it is not the “it’s just a Python loop” reading. His objection is that the humans doing the rewarding cannot evaluate what they are rewarding, and the reward signal is all the model has to go on — which makes the exposure largest exactly where the builder is least able to notice it.
Read alongside Cal Newport’s version of the same problem, the two meet in the middle: Newport argues from the training objective (LLMs produce what is plausible, not what is normative), Lopopolo argues from the rewarder’s competence. Both land on the same instruction — the risk isn’t that the system wants something else, it’s that nobody in the loop can tell whether the output was good.