Snowflake engineer Josh Reini deployed decider-2b, an open-weight Jev-style model—not TypeSafe’s Jev—inside Snowflake’s Model Registry. A companion repository supplies the custom wrapper, benchmark scripts, results, and teardown steps. This is an infrastructure test of a decision model, not a head-to-head Jev trial.

What was measured

  • Reini reports 175 correct of 231 public JevBench decisions for both batch and REST serving. That measures answers on the public set; it does not establish accuracy on private company tickets.
  • On one A10G GPU, his REST run had 103 ms median latency. Four GPUs handled a repeated-item batch at 50.2 decisions/second; that scaling run measured throughput, not newly scored accuracy.
  • At full load, Reini estimates $0.027 per 1,000 decisions for four real-time instances. But the repository states a continuously running single GPU node costs $1.14/hour, including idle time. The cheapest headline rate is not the cost of a mostly idle service.
  • A support-triage demo compares decider-2b with claude-sonnet-5 on tickets; its reported prices and speed do not include a labeled test of which model routed those tickets correctly. None of these measurements was reproduced for this capture.

The useful implementation detail

The inspected CustomModel.system_one wrapper accepts JevBench’s state and questions as JSON, returns answer JSON, and records row timing and token counts. It probes GPU attention kernels, disables torch.compile to match JevBench’s setup, warms CUDA graphs, and catches per-row exceptions as invalid answers. The batch script writes outputs and passes them to JevBench’s scoring harness; the repo pins model and harness revisions.

Operational lesson: compare throughput, startup and idle billing and held-out decision quality before replacing an existing classifier. Typed output makes integration cleaner, but a valid choice can still be wrong. Code and article inspected; no Snowflake deployment or model run performed. Related: open-model one-pass comparison.