Eduardo Vellasques, Lionel Palacin and Al Brown at ClickHouse give Jev two jobs: rank tables for an analytics question and classify failed agent runs. They then test whether feeding that ranking back to an agent improves its answers.

Ask about each table, then rank the answers

Jev receives the question and warehouse schema. One yes/no question per table asks whether answering the request needs that table. Code sorts by the returned probabilities.

On its 136-question development set over 18 tables, ClickHouse reports:

  • 0.951 nDCG@10, a measure of how well needed tables appear near the top.
  • 99.1% recall at ten: the share of reference-query tables found among the first ten suggestions.
  • Every required table appears in the first five suggestions for 84% of questions.
  • Ranking takes about 0.4 seconds per question in the authors’ setup.

The small warehouse matters. ClickHouse’s wider public-catalog pilot reports 87% recall at ten across 250 cases, with catalogs containing 75–710 tables. The ranking remains promising, but the near-perfect recall disappears.

Keep retrieval quality separate from task success

The full schema stays available. The experimental gate adds a hint when the agent has queried none of the tables Jev scores above the chosen threshold. It fires at most once per run.

Across the authors’ 20-model comparison, the hint leaves answer accuracy statistically unresolved:

  • On the development set’s 371 triggered cases, it recovers 34 answers and breaks 40.
  • On the held-out board’s 812 triggered cases, it recovers 87 and breaks 69.
  • Pooled, that is 12 net recoveries, with the authors’ 95% interval spanning −18 to +42.

The development-set reduction in query turns also disappears on held-out questions. Hints shorten some long investigations while adding work to runs that were already brief. A plausible table suggestion can change the route without improving the answer.

Let the labeler question the grader

The separate failure labeler asks for a detailed error category, a relevant prompt rule, whether a schema hint might help, and whether the answer looks correct despite failing.

ClickHouse reports 277 failures across 544 attempts in its initial development run. The labeler flags 18 possible grader errors above its selected threshold. Reviewing high-scoring cases exposed a comparator that rejected differing row counts before trying to align columns.

These flags are review leads. Typed judgments and probability scores still need correctness checks; they cannot certify either a failure diagnosis or the original answer.

Verification: read the full article and inspected the ranker, gate and labeler implementation. Four offline parsing and ranking-metric tests passed. All benchmark figures above are ClickHouse’s reported results; no live Jev calls or agent-performance reproduction was performed.