Shreya Shankar (Stanford, Bargain/LLM cascades) live whiteboards model cascades on Hamel Husain’s channel: cut LLM classification costs by ~90% with no fine-tuning and no accuracy sacrifice. 26 minutes.

The problem: the Oracle plan

  • Every record (e.g. customer support tickets) goes to one high-quality LLM (say GPT-5.6) → yes/no label (escalate to human or not)
  • Great accuracy, brutal at scale — back-of-envelope: tens of thousands of dollars for a million tickets
  • Goal: match the Oracle’s accuracy at ~95% agreement on your data, but much cheaper

The fix: a model cascade

  • Proxy model first: every ticket hits a cheap nano LLM, which returns a prediction AND a confidence score C(x) (0–1)
  • Decision point: if C(x) ≥ threshold τ, accept the proxy’s label; otherwise route to the Oracle
  • Confidence scores come free from every LLM API (token log probabilities); ground truth = “what the Oracle would say,” not absolute truth

Step 1 — find the threshold (offline)

  1. Sample 200–500 tickets, run the proxy on each → (prediction, confidence) pairs
  2. Label each sample ticket with the Oracle
  3. Simulate the cascade for every possible τ = one of the observed confidences: record accuracy and cost
  4. Return the τ with minimum cost that still meets the target accuracy — sort ascending and take the first one that qualifies
  • Trivial sanity check: τ = 1 routes everything to the Oracle → 100% by definition; the real question is how low you can go
  • Lower τ = cheaper (more accepted by the proxy) — but the proxy’s mid-confidence predictions are where it gets wrong

Step 2 — run the cascade

  • Run the proxy on ALL tickets; accept every C(x) ≥ τ; query the Oracle only on the rest
  • That’s the whole inference path

Caveats and results

  • Target accuracy on the full set is an estimate, not a guarantee — you measured it on a sample
  • Statistical guarantees are possible via concentration inequalities; their open-source package/paper (“Bargain”) adds adaptive sampling — plain random sampling wastes effort on low-confidence points that are probably wrong anyway
  • In practice: 90%+ cost cuts on a wide variety of classification tasks with just a nano model

Cascades vs. model routers

  • Routers are black boxes — no way to estimate the accuracy you’re actually getting
  • Cascades are sample-driven and fit to YOUR data; the threshold search tells you whether any τ even meets the target
  • Works for binary AND multi-way classification (well-calibrated confidence from logits over the class tokens); not for open-ended tasks — poorly calibrated probabilities there often mean no τ exists
  • Don’t trust the nano model’s confidence a priori — that’s fine. The simulation either finds a threshold that meets the target or it doesn’t; “the procedure is not going to lie to you.”

“You don’t have to have any confidence in the confidence.” — Hamel Husain, on why the simulation matters more than trusting the nano model