Marcus Rox’s jev-tracker is more than another leaderboard screenshot. It freezes a 75-case reranking benchmark, commits the raw model answers and derived measurements, and makes every displayed number traceable to a repository artifact.

I inspected the repository and its implementation files but did not rerun the GPU experiments. The included offline test suite emitted all expected progress dots in focused runs, but the pytest process did not terminate before the execution timeout, so I am not treating that as a clean independent test pass.

What it measures

  • Each case starts with a parent query, candidate children, and relevance labels.
  • A decision model scores the children through a common System One request-and-response contract.
  • The main quality measure is kept mass at k: how much labeled relevance remains after keeping the top 50, 100, 150, or 200 candidates.
  • The repository records quality, warm-run cost, latency, model family, serving method, runtime, GPU, and query count separately.
  • Raw answers, run configurations, calls, costs, latency records, and generated tables live under data/experiments/ rather than disappearing behind the website.

The current result

  • The committed 75-case table reports Jev Score at 0.893 kept-mass@50 and 0.961@200.
  • Kev-27B Score is the nearest listed open alternative at 0.887@50 and 0.954@200.
  • The older production pipeline—GPT-5-mini with three listwise passes and reciprocal-rank fusion—records 0.711@50 and 0.837@200.
  • Those are the project’s measurements on one frozen reranking task, not a general accuracy ranking for decision models.
  • I verified that the numbers appear in the committed result JSON and Markdown; I did not reproduce the Modal runs.

Why the harness is useful

  • Every scorer is normalized into SystemOneRequest, SystemOneResponse, and model-independent raw records.
  • The contract makes it possible to compare Jev, Kev checkpoints, Laya checkpoints, hosted APIs, and model-specific adapters downstream without special-casing the evaluator.
  • Validation requirements specify exact Jev and production reference values and a tolerance of 0.002 for selected Kev reruns.
  • Cost and latency are not inferred from model marketing. The harness records run timing and GPU billing inputs, while explicitly excluding CPU and egress from the cost model.
  • The generated site links displayed values back to their source JSON, which makes a surprising number auditable rather than merely impressive.

Discovery is part of the system

The repository also implements the work normally left outside a benchmark:

  • A crawler searches GitHub, Hugging Face, arXiv, the web, X, Hacker News, and the maintainer’s Slack workspace.
  • Candidates are deduplicated, then classified as runnable, requiring an adapter, or unrelated.
  • Runnable models become experiment configurations; models needing new integration work can be proposed for approval rather than silently skipped.
  • A six-hour automation runs approved work, commits triage decisions and artifacts, regenerates the site, and leaves an inspectable history.
  • The current summary says 19 proposals await approval, making the queue and its omissions visible.

Important caveats

  • The benchmark is only 75 cases from one parent-child reranking workload. It cannot establish broad decision quality, calibration, safety, or usefulness in another domain.
  • Jev’s committed answers are the reference run; Jev itself is not re-called by the public harness without an API key.
  • kept-mass@k measures retrieval preservation, not whether a downstream decision is correct.
  • Cost comparisons depend on the chosen hardware, concurrency, cache state, and the repository’s accounting rules.
  • Some latency rows deliberately distinguish ordinary warm service from a cache-warm fan-out experiment; collapsing them would create a misleading leaderboard.
  • A typed response only establishes that a model produced a valid structure. It does not establish that its scores rank the right candidates.

Operational inspection

  • Source inspected at commit 18a19b382c239a39843b55ee3839466dbf8c701f from October 3, 2026.
  • The repository contains the benchmark data, raw compressed answers, adapters, experiment lifecycle code, crawler, queue, static site, and tests described in the specification.
  • I ran focused offline pytest selections. They printed 24 and then 43 passing-progress dots in separate attempts, but both processes remained alive until the 120- and 300-second execution timeouts. Because pytest never returned a success status, this note does not call the suite independently passed.
  • No GPU experiment, hosted API benchmark, or security audit was performed for this capture.

Takeaway

The important contribution is not that Jev currently leads this table. It is the separation of claims into committed inputs, raw outputs, derived metrics, operating assumptions, and automation decisions. That structure makes it possible to challenge the benchmark without first reverse-engineering it.