Jina Yoon describes how PostHog uses decision models, which return a choice or score, to handle narrow judgments around generative AI. The strongest examples pair small checks with explicit decisions about which mistakes cost more.
- Two personal-information checks trigger a rewrite at a score of 0.4. PostHog reports that 98.7% of session summaries skip that generative-model call.
- Its Slack bot needs two scores of at least 0.85 before speaking uninvited. Unnecessary replies are the costly mistake there.
- Checking search results for relevance improved ranking quality by about 25% on 20 test queries, according to the team.
- Ranking code-review findings produced an inconclusive AUC of 0.52, a measure where 0.5 is chance-like. Large chained requests also overwhelmed a shared model server.
PostHog found that thresholds changed between model versions. It kept one judge running without acting on its decisions while recalibrating against human-labeled examples.
Caveat: these are PostHog’s reported results; skipping rewrite calls does not establish how much personal information the checks miss.
The 71-comment thread on Hacker News challenges the title and asks what comparisons the article leaves out.
What the thread adds
- mike_hearn argues that Jev-like decision models are still language models, optimized for one-shot selection rather than generating tokens. The interface change matters more than the headline’s category split.
- throwawayffffas questions whether a router can choose among newer models using only short descriptions of their capabilities. The article supplies no routing-accuracy result.
- nostrebored asks why use a larger, remotely hosted decision model rather than train a dedicated search reranker. Their comment offers a comparison to test, not measurements settling it.
- aabdi takes the opposite position on cost-cutting: use a strong model unless you can build adequate evaluations, because weak tests can hide poor real-world performance.
- TacticalCoder questions the article’s conversion of a 0.85 score into an error rate: “how do we know it’s 0.85 and neither 0.75 nor 0.9?” The article qualifies its opening example with “In a calibrated model”. Applying that assumption to a production task needs validation beyond choosing a threshold.
Handles are pseudonymous; HN publishes no per-comment scores, and ordering is HN’s ranking. This is a slice of the thread, not a consensus.