Amplitude’s Vinay Goel and Ram Soma tested Jev 1.13 as a first-pass monitor for product analytics: scan many charts cheaply, find recent behavior worth investigating, and escalate the small interesting set. Jev found every injected exception in their synthetic benchmark, but only 55.8% of its alerts were true positives.

That tradeoff is the result. A cheap detector can make broad screening practical, while false positives can move the cost into the expensive investigation stage. I reviewed Amplitude’s writeup and result figures; I did not reproduce the paid model calls.

The test

  • The authors created 64 synthetic charts containing 190 time series across event segmentation, funnels, retention, and customer journeys.
  • They injected meaningful recent changes into 24 charts; the other 40 were realistic controls that should not be escalated.
  • Roughly 15% of the underlying series included spikes, drops, sustained level changes, segment divergence, volatility changes, or possible instrumentation breaks.
  • Each model received chart metadata plus CSV-formatted time-series data and answered one bounded question: whether a meaningful recent exception deserved deeper investigation.
  • Jev 1.13 was compared with Luna, DeepSeek V4.1 Flash, GLM-5.3 Flash, and Sol.

This is a controlled synthetic test, not a production estimate. Amplitude says a real deployment should use blinded samples of actual charts labeled by analysts or chart owners.

Perfect recall came with modest precision

  • Jev recorded 100% recall: it caught all 24 charts containing injected exceptions.
  • Its precision was 55.8%, meaning a large share of the charts it escalated were controls.
  • Amplitude reports median latency of 0.14 seconds and cost of roughly $0.06 per 1,000 charts.
  • In this test, Jev was less than half the cost and less than a quarter of the latency of DeepSeek V4.1 Flash, the next cheapest and fastest comparison.

Those numbers are Amplitude’s results on its own synthetic data. They were not reproduced here, and the article does not make the complete benchmark harness available in the source page.

False positives belong in the cost model

For “manage by exception,” high recall can be the right bias: missing a conversion collapse or instrumentation break may cost more than reviewing a normal chart. But the first-stage API bill is not the whole system cost.

  • If every flag launches an expensive root-cause agent, low precision multiplies the downstream model spend.
  • If every flag reaches a person, review capacity becomes the limiting resource.
  • Raising the threshold may improve precision while hiding exactly the subtle failures the monitor exists to find.
  • The correct threshold therefore depends on review capacity and the relative cost of missed versus unnecessary investigations.

A cascade handles uncertain cases. It does not automatically catch confident errors, especially when the cheap and expensive models share the same ambiguous schema or conceptual blind spot.

The stronger lesson is to observe outcomes

Amplitude’s argument goes beyond this benchmark: model traces contain the decision but often not the eventual evidence that reveals whether it worked.

  • A ticket routed with 0.94 probability may look correct in the trace and reopen two days later.
  • A generated artifact may receive no explicit rating but correlate with whether the user saves it or returns.
  • Those product events are noisy, but they can expose systematic failures that another model judge cannot see from the original exchange.
  • Randomized holdouts, segmentation, and sufficient volume can make those signals more useful than treating any one event as ground truth.

Amplitude’s product recommendation is self-interested—the article promotes Agent Analytics—but the underlying point stands: a decision layer should be linked to delayed outcomes, not graded only by its own confidence or by another model’s opinion.

What a production version needs

  1. Build a blinded set of real charts with labels from people who own the metrics.
  2. Measure precision and recall by chart type, not only in aggregate.
  3. Price the entire route: screening, verification, deep investigation, and human review.
  4. Pin the model version, prompt, schema, and threshold that produced the result.
  5. Audit accepted cases as well as escalations so confident misses remain visible.
  6. Join decisions to later product outcomes, while treating those outcomes as noisy evidence rather than automatic truth.

Takeaway

Jev looks useful here as a high-recall screening layer, not as an autonomous analytics verdict. Amplitude’s small benchmark shows that its speed and price can make continuous monitoring feasible. It also shows why cheap decisions are not cheap if weak precision sends too much work downstream.

Typed output only guarantees a valid answer shape. The operational question is whether the whole route catches enough real exceptions at an affordable false-positive rate—and whether the system can discover when a confident decision failed later.