Search-relevance engineer Doug Turnbull’s counterintuitive trick for LLM classification into a huge fixed taxonomy (the Wayfair WANDS dataset has hundreds of legal category paths like Furniture / Living Room Furniture / Coffee Tables & End Tables / Coffee Tables).

The problem

  • Constraining an LLM to a legal vocabulary via structured outputs means shipping a giant Literal[...] of ~500 values in a Pydantic schema with every call — expensive, and there’s an upper limit on what you can send.
  • Classic approach works, but doesn’t scale cheaply with small/dumb models.

The trick

  • Don’t constrain — hallucinate. Ask a cheap LLM to invent plausible, never-seen classifications for the query (brown coffee table → something made-up like Furniture / Living Room / Tables / Coffee).
  • Resolve with embeddings. Precompute an in-memory MiniLM embedding for every real classification, embed the fake one, and dot-product to find the nearest real category — which lands exactly on Furniture / Living Room Furniture / Coffee Tables & End Tables / Coffee Tables.
  • Why it’s cheap: hallucination tasks run on small/dumb models, and the schema never goes over the wire.
  • Code: a Colab notebook plus cheat_at_search/enrich/vocabulary.py on GitHub.