James Carr inspected his Oh My Pi session logs from September 30 through October 11. His useful finding is narrower than the broad Jev pitch: almost all recorded usage was semantic code search, with a smaller role choosing subagent thinking effort.

I read the complete article. The usage, costs, timings, scores, and project outcomes below are Carr’s reports, not independently reproduced results. His raw session corpus is not supplied in the article.

Search first, judge a bounded candidate set

  • Ordinary grep pre-ranks files; Jev judges filenames, excerpts, then passages rather than searching the whole repository unaided.
  • Carr reports 61 searches making 604 Jev requests, with 757 ms median wall time per search.
  • In his email-helper example, the intended file scored 0.94, while a near miss scored 0.22 against a 0.20 threshold.
  • In an empty /metrics investigation, search ranked the router and metrics implementation above configuration; Carr says a fresh node simply had no events yet.

This is evidence of a concrete retrieval workflow—not a labeled relevance benchmark. Displaying weak matches helps the next agent inspect them; it does not make those scores calibrated correctness probabilities.

Separate actual usage from demonstrations

  • Carr reports 629 total calls and $0.0888 in Jev usage cost; this excludes the surrounding coding models and total development cost.
  • The smaller categories were 19 automatic effort selections and six judge() calls.
  • Five of those judging calls came from one cell using invented inputs, including early-stop and unsupported-test-success checks.
  • Carr explicitly says those two checks had not run on Jev in his actual sessions.

The article’s rounded cost rows sum to $0.0889, not its printed total; that could be rounding, but the underlying billing data is unavailable. Its effort breakdown also leaves three of the reported selections unspecified.

What to take into your own harness

Log purpose, candidate counts, wall time, input usage, and downstream outcomes separately. Compare the search against plain grep on labeled tasks before assuming that model scoring improves retrieval or choosing a threshold.

Typed output is not a correct decision. This is a valuable operational account of where calls go, not evidence that Jev reliably catches agent failures or optimally allocates reasoning effort.