GraphReflex uses Neo4j to retrieve products and TypeSafe Jev to label query words, select search directions and judge candidates. It compares that pipeline with a Claude Opus 5.5 agent using graph tools and a plain vector-search baseline. Its public showcase replays saved runs; clicking a race does not make fresh model calls.

Compare the right product, not just a valid choice

  • On 81 human-verified STaRK questions, the project’s latest run reports a correct product first for 70.4% of GraphReflex searches and 61.7% of Claude searches. The paired difference is inconclusive (p = 0.09).
  • On 379 human-labelled WANDS queries, it reports 75.7% versus 77.3%. That difference is also inconclusive (p = 0.52).
  • The project reports median retrieval times of 4.2 versus 13 seconds on STaRK and 2.2 versus 8.0 seconds on WANDS. Its listed model-call costs are $1.93 versus $68.03 and $1.03 versus $37.36 per thousand searches. These are the author’s recorded runs, not independently reproduced measurements.
  • Vector search is much faster and has negligible model-call cost in this accounting, but places the right product first less often in both tests.

The code fetches product records and, when a query names another product, supplies relevant bought-together or viewed-together links to Jev’s yes/no judge. An earlier version found linked products but hid the links from that judge. The author’s error analysis traced failures through retrieval, judgment and final comparison before revising the pipeline.

The score threshold is 0.65 in the illustrated run. The repository also calculates calibration against labelled outcomes. A probability attached to a typed answer still needs checking against the actual product match.

Do more searches help?

The project tested up to two extra search rounds. It reports essentially unchanged first-result accuracy on STaRK and lower first-result accuracy on WANDS, with higher costs. The default therefore uses one search pass.

Caveat: the author revised GraphReflex after studying the same questions, while Claude’s saved run stayed fixed. The 11 November repeat fell below the previous day’s GraphReflex result. A held-out comparison is still needed.

Verification: inspected the README, Jev client, directed retrieval and judging code, paired-test implementation, and saved-results documentation. I did not run Neo4j or the paid benchmark, reproduce its performance figures, or conduct a security audit.