romain-cll’s reflex-rag tests Jev as the judge inside a document-retrieval loop. Its report connects measured results to saved traces, failed approaches and a small policy change.

RAG means answering from documents retrieved at question time. Here, Jev decides which notes to keep or follow; Claude Haiku writes the final answer.

Compare the same retrieval job

The author tests a synthetic company vault containing 202 Markdown notes, with separate tuning and test questions. Each configuration runs the 60-question test split three times.

  • The Haiku-judged loop and Jev-only loop both average 51.7 correct answers out of 60, with 86.8% complete contexts on answerable questions.
  • The author reports retrieval p50, the middle response time, falling from 6.45 seconds to 0.68 seconds. Estimated retrieval cost falls from $0.00203 to $0.00085 per question.
  • Offline recomputation matched the reported counts, nearest-rank timing summaries and costs from the saved traces. These are recorded API timings and token-price estimates, without fresh model calls.
  • Against simple top-five retrieval, Jev’s loop improves average correct answers from 42.7 to 51.7. Its estimated total cost remains about 2.8 times higher.

The headline speedup describes retrieval. The author’s end-to-end p50 is 2.52 seconds for Jev’s loop and 2.23 seconds for simple retrieval.

Ask whether the fallback can still help

The initial fallback sent nearly accepted notes to Haiku even after Jev had kept a note judged to contain the answer. Offline inspection reproduced 83 such calls out of 87 across the author’s three test runs.

  • The replacement rule asks Haiku only while the retained context has no answer note.
  • Across those saved runs, fallback calls fall to five, all before an answer note is retained.
  • The author reports retrieval p50 falling from 2.24 seconds to 0.70 seconds, with complete contexts unchanged at 88.9%.
  • Correct answers average 52.7 before and 53.3 afterward. That small difference sits within the observed run-to-run spread.

The implementation checks retained notes from earlier turns and Jev’s current batch. Keeping a stepping-stone note alone leaves the fallback available.

Keep conflicting evidence visible

Most of the measured gain comes from questions that require following links between notes. Simple retrieval already performs well on the other categories.

The report also documents a failure: Jev’s cross-note veto can discard one side of a real contradiction. Typed verdicts keep the output well formed; they can still remove evidence the answer needs.

Caveat: this is a small, Claude-written synthetic corpus and one Haiku comparison. The experiment supports this retrieval design, not a general quality-equivalence claim.

Verification: inspected the report, Jev HTTP adapter, judge, fallback, policy and grader; regraded 900 saved answers and recomputed metrics across 15 test runs. The local mocked suite passed 1,862 tests. No live model evaluation was rerun.