WebSpeaker tested Jev in the planning stage of its website-support chatbot. The useful result is negative: a fast decision component did not make the product faster, because changing the planner also changed the work downstream.

What WebSpeaker tested

  • The pilot used synthetic Polish and English questions; WebSpeaker says no customer data was sent to Jev.
  • A 60-question isolated routing test asked whether to retrieve documents, answer directly, or hand off to a person.
  • Three end-to-end variants then replaced Gemini decisions with Jev, put Jev before Gemini, or ran the two in parallel.
  • The broader pilot covered eight scenarios repeated three times, so this is engineering evidence—not a definitive benchmark.

The isolated result looked promising

  • WebSpeaker reports both models at 96.7% on the 60-question routing test.
  • One Jev decision took a median 232 ms, while Gemini’s larger planning task took 1,824 ms; these were explicitly not equivalent workloads.
  • In the integrated tests, Jev selected the expected decision 91–92% of the time versus 87% for Gemini.
  • These are WebSpeaker’s measurements. I reviewed the writeup but did not reproduce its model calls.

The complete chatbot got slower

  • Putting Jev before Gemini raised Jev’s median latency to 872 ms as more decisions were added, and the full response arrived about 27% later on average.
  • Running the models in parallel cut the analysis stage by 9% for a typical request and 24% in slower cases.
  • Even then, the final answer arrived about 22% later, and one test produced an internally contradictory response.
  • Parallel components lacked shared context: Gemini could prepare a search query without knowing whether Jev had decided that retrieval was necessary.

Why local optimization lost

  • Jev changed routes, retrieval breadth, and whether an extra model check ran—not merely the duration of one existing call.
  • The reported 151 ms saved in analysis was consumed by retrieval, extra validation, generation, and answer checking.
  • A typed answer guarantees that routing output fits the allowed schema. It does not guarantee that the route is correct or compatible with the rest of the plan.
  • Component latency therefore cannot stand in for user-visible latency; the unit of evaluation has to be the whole conversation.

The caveats are part of the value

  • The sample is small, final-answer quality was not systematically scored, and WebSpeaker did not finish a cost comparison.
  • The article does not establish that Jev generally slows chatbots. It establishes that this particular decomposition did not improve this pipeline.
  • WebSpeaker did not deploy the integration and says Jev had not reached its production data-handling review.
  • A stronger follow-up would freeze routing semantics where possible, label end-to-end outcomes, and measure latency, quality, checks, retrieval work, and cost on the same requests.

“The speed of one component tells you nothing about the speed of the whole system.”

The practical lesson is broader than Jev: replacing one planner with two models changes information flow and system behavior. Benchmark the path the user waits for, not the fastest box in the architecture diagram.