LangChain’s Sydney Runkle and Eugene Yurtsev put a Jev-based model router inside Open SWE, their coding-agent harness. Unlike a standalone routing demo, they compared live work against a single-model control. The router picks a tier from the thread’s first message and uses that model for the whole thread.
How they chose
- They grouped a week of agent requests into task types using an LLM, then set criteria for three cost/capability tiers: GLM-5.3-Flash, GPT-5.6 Sol and GPT-6 Astra. Those category labels are heuristic, not a ground-truth difficulty measure.
- Jev selects among those tiers in middleware. A typed choice says which model to try, not whether the eventual code is good. LangChain says replacing an LLM classifier with Jev made the classification stage nearly 50× faster; I did not reproduce that claim.
The live comparison
- LangChain reports an A/B test across 973 threads, splitting traffic between the router and always using GPT-6 Astra. 29.2% of routed threads produced a merged PR versus 27.3% of control threads (p = 0.49). The measured difference was inconclusive—not evidence that the two are equivalent.
- Median attributed LLM cost was $0.94 routed versus $2.61 control, a reported 64% reduction. Mean and upper-tail costs also fell in LangChain’s results. This is the team’s own workload and measurement, not independently reproduced; choosing the most expensive model as the baseline matters.
- A separate comparison against always using the fastest model was stopped early after engineers reported poor output. It was not large enough to quantify the difference.
The boundary
A merged PR is a useful real-world signal, but it does not measure hidden bugs or review effort. The router also misses changes of task halfway through a conversation; switching then could discard a model’s prompt cache. The useful pattern is to track end-to-end outcomes alongside cost and use a baseline that reflects your actual alternative—not infer quality from the router’s tidy output. Related: a cache-aware coding-agent router.