Baldur Bjarnason wrote this essay in July 2023; it resurfaced on Hacker News this week. His question is why so many people come away from a chatbot convinced they have been talking to something intelligent, when nothing in a model of language should produce that impression. His answer: the illusion lives in the user, and it works exactly the way a psychic’s cold reading works — by accident.

The con, and its six stages

The essay’s core move is a side-by-side. The psychic’s con has stages, and each has a chatbot equivalent.

  • The audience selects itself. People who don’t believe in psychics don’t buy tickets. People who don’t believe in “AI” don’t use chatbots, so the room is already open-minded.
  • The scene is set. Hype primes expectations. Vendor warnings do double duty: “it’s early days” and “hallucinations” both describe the model as a person having a bad day.
  • The prompt filters. Most users take the first answer and leave. Some start a conversation instead.
  • The mark tests themselves. The reply sounds specific to you but is statistically generic. The model has read so much text that “a plausible continuation of your prompt” feels like insight.
  • The loop. The mark asks more, the replies keep landing, and conviction grows. Bjarnason calls this reinforcement learning for the mark.
  • “Wow, it thinks.” The user is left with the sense of a proto-mind.

The psychology it leans on is subjective validation: our minds treat a statement as accurate if it has personal meaning for us. That is the engine behind Forer (or Barnum) statements like “you tend to be hard on yourself” — empty, but most people accept it as a portrait of themselves.

Bjarnason collects the whole family under one term, validation statements, and lists the tactics:

  • Forer/Barnum statements — generic truths that feel specific.
  • The vanishing negative — “you don’t play the piano?” reframes as a hit whether you say yes or no.
  • The rainbow ruse — “you’re calm, but you can get very angry if provoked” covers both poles.
  • Statistical and demographic guesses — “you have a scar on your left knee” applies to nearly everyone.
  • Unverifiable predictions — unfalsifiable, so it can never be wrong.
  • Shotgunning — fire enough statements and the mark remembers only the one that landed.

Why the fine-tuning made it worse

The most mechanical part of the argument is about how raw models are turned into chat assistants. The standard method is reinforcement learning from human feedback: humans rank several candidate replies, those rankings train a separate scoring model, and that scorer nudges the chatbot toward the higher-ranked outputs.

The catch is that a ranking signal carries no facts. A scorer that is itself a language model cannot reward a reply for being true — only for having the tone, style, and structure that human raters associated with replies they rated as accurate. Bjarnason adds that the raters are low-paid contractors with neither the time nor the topic expertise to fact-check what they rank.

His conclusion: the optimization target became validation statements. “In trying to make the LLM sound more human, more confident, and more engaging, but without being able to edit specific details in its output, AI researchers seem to have created a mechanical mentalist.”

The edges of the claim

Two parts are worth keeping separate from the argument’s rhetorical force.

  • Susceptibility has nothing to do with intelligence. The essay argues the effect gets stronger for people who consider themselves smart, because they apply their intelligence to rationalizing the trick rather than noticing it.
  • Bjarnason is explicit that he does not think this was deliberate: “I personally think that’s extraordinarily unlikely.” His reason is unflattering to both sides — “the tech industry just isn’t that good at software. This illusion is, honestly, too clever to have been created intentionally by those making it.”
  • He also lists what he considers the same superstition in other clothing — hooking a model to an MRI and claiming mind reading, inferring criminality from faces or gait, faith healing, prophecy, and using models to judge a job candidate’s honesty.
  • His advice is blanket: don’t put an LLM in your product, website, or organizational process. The essay ends with a pitch for his own $35 book on the topic, which the thread noticed and did not let pass.

The 238-comment thread on Hacker News is mostly an argument with the premise rather than with the mechanics — and the disagreement itself is the interesting part.

What the thread adds

  • bonoboTP — the top-ranked objection is to the question rather than the answer: “I don’t care if it’s ‘intelligent’ … If it generates functional output that works, then it works. It’s not a psychic’s con when it outputs Lean-verified proofs. It isn’t a con when it can find and exploit zero-days.” Their framing is that the essay sits in a stage of grief the field has moved past.
  • krupan — the counter to that counter, on the same example: the model did not solve it, “humans did, using a statistics-based computational tool,” and we don’t know how that tool was used, whether it is the same one available to everyone, or what it cost in money, energy, and time.
  • azakai — the mechanism objection, replying to a commenter who said the premise was made up: neural networks were designed from the 1950s onward as models of brain computation, and “if you are not a dualist, then computation is what the human brain does.” In their telling the author is unaware of the field’s own history.
  • bunderbunder — a correction on Turing: the imitation game’s real bar was an interrogator unable to do much better than chance across many trials, not being fooled once. Turing “explicitly considered deliberate deception as an essential part of the test,” so passing it is harder than the thread’s shorthand suggests.
  • natbennett — accepts the pushback and keeps one line: “Lots of comments talking about how recent accomplishments disprove the article but I think this bit holds up pretty well” — the essay’s claim that delegating your judgment to a chatbot is functionally like phoning a psychic. Their summary: “LLMs are very good at tricking people into thinking they have capabilities that they don’t.”
  • trescenzi — why the naming fight is practical, not philosophical: if models are treated as independent intelligent entities, blame lands on them instead of on their users — relevant “when some of the most powerful people are looking to absolve their crimes by passing them off on their LLMs.”
  • vhantz — the middle position, aimed at both sides: “Those who think LLMs are intelligent don’t know enough about models. And those who think they are useless don’t know enough about models.”
  • mstank — the incentive read of the essay’s ending: it is against the technology but closes by selling the author’s own book, so “I can only understand it with the context that the writer is trying to sell their book.”
  • emaro — names a tension worth watching in the thread itself: the same discussions praise LLMs for solving hard problems and call them “just a tool.” If a tool implies no agency, and intelligence implies agency, can there be an intelligent tool?

The standing complaint the essay never answers

Five separate commenters arrive at the same correction: the piece is from July 2023. bonoboTP adds the note as an edit (“Then (2023) should be added in the submission title”), daishi55 edits theirs (“oh I see, written in 2023”), mstank comes back to say “Just realized this was written in 2023. Can’t believe how wrong we were about AI back then,” ryuuseijin gives it as “The post makes sense for 2023. Today, not so much,” and NinjaTrance posts the date on its own. One commenter states the actual ask that none of them can satisfy from the thread: daishi55 again — “I wonder if the author has updated their attitude towards this question? Indeed that would be the most interesting thing to know.”

That is the same fight as the bearish-on-LLMs-after-Navier-Stokes post, from the other side: one writer says the headline wins are the best case that will ever exist, and this thread’s majority says the headline wins are proof the 2023 argument is obsolete. Neither side here produces a method for telling the two apart.

The comment summaries above come from the HN thread, not from the essay, and vice versa. Handles are pseudonymous, HN publishes no per-comment scores, and the ordering is HN’s own ranking — so this is a slice of the thread, not a consensus.