Glean evaluated Jev against four existing enterprise workloads instead of treating “decision model” as one undifferentiated use case. The result is the useful kind of mixed: one clear routing win, two losses to specialized systems, and one promising consistency result that still lacks human accuracy labels.

All figures below are Glean’s offline measurements. I did not reproduce the private datasets, calls, or latency tests.

Query classification favored adaptation

  • Zero-shot Jev agreed with Glean’s LLM-based broad-task baseline on 66.8% of cases.
  • A base Laya model reached 35.9%, while a locally fine-tuned Laya reached 74.5%.
  • Glean reports about 12 minutes for Jev at concurrency four versus about 90 seconds for both Laya runs on a developer machine.
  • Agreement with an existing model is only a proxy for correctness, but the experiment makes the tradeoff concrete: Jev buys immediate flexibility; labels and fine-tuning can buy better task-specific performance.

Routing was the strongest Jev result

  • Glean replayed 751 golden examples through a three-expert router and compared both systems with the recorded labels.
  • Jev improved precision and recall for all three expert routes while also improving the dominant “no expert” route.
  • On 40 positive transfer calls, Glean measured a median 8.1× latency speedup over its existing LLM router.
  • This is still an offline test, and a non-transfer adds a Jev call before the main model starts working. Glean says that blocking tax needs live derisking.

Search ranking exposed the limit

  • On 4,855 paired search queries, Glean’s production order reached 51.18% Recall@6; Jev Choice reached 40.21%.
  • Jev cost roughly $0.00044 per query and took 0.195 seconds at p50, according to Glean.
  • It gave identical scores to about 37 of 41 candidates in an average query. Reusing production order to break those ties lifted Recall@6 to 44.3%, but that gain did not come from Jev’s ordering.
  • Typed probabilities made the output valid; they did not make the ranking competitive with the specialized production stack.

Citation judging was repeatable, not validated

  • On one fixed 1,448-word response, Glean measured 6.6 seconds and $0.014 for Jev, versus 81–86 seconds for GPT-5.6 Luna without reasoning and 227–260 seconds with xhigh reasoning.
  • Across three identical runs over 28 paragraphs, Jev changed one paragraph-level recall verdict; the Luna configurations changed seven and five.
  • Glean explicitly does not conclude that Jev was more accurate: the judges used different criteria, and the experiment had no independent human labels.
  • The result supports a narrow claim—Jev can apply one citation policy cheaply and repeatably—not the broader claim that its verdicts are correct.

“Jev does not change the fact that classifiers already existed. It makes a good zero-shot classifier much easier to use.”

The practical rule is to benchmark the exact bounded decision. Stable, high-volume labels still justify a fine-tuned classifier; open-ended arguments still require a generative model; and every confidence threshold needs real workload labels before it controls production actions.