TJ Eastmond’s Scope uses static code analysis to shortlist source chunks, Jev to judge their relevance, and deterministic code to assemble the final context. Its useful contribution is a traceable selection pipeline with an explicit offline baseline—not a claim that a classifier can search an entire repository unaided.

I inspected the complete README, orchestration, candidate selection, Jev adapter, response validation, selection/support logic, and evaluation documentation at revision cfe1a28fb9e0ad4bcdfa0ea96307289199a5ea7f. I did not run Scope, replay its evaluation, or call Jev. This was source inspection, not a security audit.

Bound the model’s job

  • Static retrieval shortlists at most 40 chunks under the inspected configuration.
  • Jev receives the task, source locations, metadata, and up to 6,000 characters per candidate for a yes/no relevance question.
  • Application code keeps relevance scores at or above 0.5, then adds supporting declarations and merges overlapping regions.
  • Final output retains full selected chunks; there is no hard output-token budget.
  • Failed or malformed provider responses fail the run rather than silently switching to the offline baseline.

These are inspected implementation settings, not independently measured performance results. Code absent from the shortlist is not judged; code beyond a truncated candidate’s prefix is not directly assessed.

Read the paired result, not just the compression headline

The pinned repository’s generated report records 27 tasks across five fixtures, including 11 held-out tasks. Its maintainer-recorded results are:

  • Jev: pooled required-chunk recall 15/20 (75.0%), mean precision 54.8%, mean source-character reduction 90.2% against eligible repository text.
  • No Jev: pooled recall 16/20 (80.0%) and mean precision 11.9% on the same held-out tasks.
  • The held-out logging task loses Logger.info after Jev filtering; one required chunk out of two remains.

Those figures were not reproduced here. The result is more selective context with a recall tradeoff, not proof of better patches or lower total development cost. Character reduction is not a measured token or billing saving.

The README still says 26 tasks, 54.6% precision, and 90.6% reduction. The figures above follow the generated report in the pinned checkout instead of mixing snapshots.

Keep the evidence boundary visible

  • The evaluation can replay committed Jev judgments offline, but replay cannot independently validate the original model calls or recorded latency.
  • Its provenance records the moving jev-latest alias, not an immutable response model ID.
  • SCOPE_JEV_PAYLOAD=print offers a no-send payload preview; normal operation sends shortlisted source to TypeSafe.
  • Maintainer-documented secret filtering is best-effort, not a guarantee that committed source is safe to transmit.

Typed output is not a correct decision. Scope’s most reusable lesson is to measure what filtering removes—including required context—and retain the baseline that makes that loss visible.