Logan Markewich’s jeff serves the TypeSafe System One request format from a self-hosted GLiFormer model. An application can point the official SDK at its endpoint for typed yes/no, choice and score answers. The same interface does not make its judgments interchangeable with hosted Jev.

The implementation

  • The code turns each question into a group of candidate labels for a classifier. By default it isolates yes/no questions into separate encoder passes; choice and score questions can share one.
  • That isolation prevents other questions from changing a yes/no result, at the cost of encoding the state again. Backend options include PyTorch on GPU or CPU and an ONNX encoder for CPU.
  • The server has optional bearer authentication, request limits and batching. Authentication is off by default if no keys are configured.
  • Its displayed confidence is a normalized gap above a uniform guess, not the chance the answer is correct. The source says displayed probabilities and score calculations can differ under the default temperature.

The measured tradeoff

The repository cites an official JevBench v1.2.2 run by the benchmark maintainer: jeff scores 66.9 (#9 of 18) versus Jev 75.3 (#2) on the composite. On the hard tier, jeff gets 37.7% correct versus 74.1% for Jev. The author’s own same-item public-tier run reports 37.8% versus 72.1% on 111 hard questions. None of those runs was reproduced for this post.

Markewich estimates a lower cost per request on a well-utilized Modal GPU, but also reports an idle-container floor and a throughput limit at its web ingress. That estimate is not a guaranteed deployment bill; the benchmark cost score uses another pricing assumption. The errors matter more than API compatibility on policies, ambiguous requests and other high-stakes decisions.

I inspected the README, grouping/engine code and benchmark report at revision 34b32f99a727c47b679adde33f4702a001e02979. I did not run inference or the test suite. This is not a security audit. Typed output is a contract for the caller; decision quality still needs labels, calibration checks and a review policy for the actual workload.