Tech With Tim tests Jev against Gemini 3.1 Flash Lite and GPT-5.5 on routing and command screening. The video is sponsored by OpenRouter; these are the presenter’s reported experiments, not independently reproduced measurements. I reviewed the complete transcript, not the datasets or execution logs.

The bounded-decision case [07:43, 14:55]

  • In 50 presenter-created support tickets, Tim reports Jev classified 49 correctly, Gemini 48, and GPT-5.5 41. He also notes that the correct labels of borderline tickets are debatable.
  • Batching 50 decisions into one Jev request lowered the reported price, but batch-versus-single-request comparisons need their harness and workloads spelled out.
  • All three models obeyed the requested one-word format in this comparison: a typed response is convenient, not evidence that the competing models cannot produce valid structured output.

The safety failure is the useful result [09:52]

  • Tim generated command variants, including Base64 and hex encodings. He reports Jev missed some risky encoded commands and wrongly blocked many harmless ones; his initial overall result was 91% correct on that set.
  • Decoding the payload in ordinary code before asking Jev raised his reported result to 98.4%. This shows why preprocessing and explicit policy belong outside the classifier. Even the improved score is not authorization to execute an arbitrary command: review, sandboxing and deterministic restrictions remain necessary.

Banking77 and escalation [16:12, 18:24]

  • On a presenter-selected, balanced subset of roughly a thousand Banking77 customer messages across 77 intent labels, Tim reports Jev at 80%, Gemini at 78.5% and GPT-5.5 at 84.9%. He reports lower Jev cost and latency; neither the exact sampling nor the model outputs were independently checked here.
  • Tim reports Jev’s ≥0.95-confidence bucket was correct 95.8% of the time on this subset. That does not establish calibration for a new inbox or policy.
  • Escalating low-confidence cases to GPT-5.5 reportedly improved the same set to 83.7% at a 0.70 cutoff. Choosing and judging a cutoff on the same data risks an optimistic result; verify on held-out traffic before automating action.

“Once things were encoded in B 64 or hex, it clearly couldn’t actually read them.” — Tim [12:10]

The transferable design is a fast proposal followed by validated thresholds and a safe fallback—not a probability treated as ground truth.