Brian W. Zhang’s askjev is a large-scale attempt to map how a decision model answers closed questions, rather than rank it against another model. The repository exposes the pipeline, experiment methods, and code—but not the answer dataset or experiment results.

The method

  • According to the project README, Jev answered 1,109,409 questions from 312 sources; 1,043,973 appear on a topic map. These are author-reported totals, not counts I reproduced.
  • Source adapters preserve answer keys and human response distributions when available. Questions pass through screening, embedding-based deduplication, Jev topic placement, answer collection, and measurement.
  • The project asks questions again with reordered options or reversed scales to detect instability. Some questions ask Jev to predict how most people would answer, which is distinct from Jev’s own answer.
  • src/askjev/jev.py hashes requests, caches responses, logs sent calls, and handles rate limits; place.py implements a bounded beam walk through the topic tree with a faster embedding shortlist for large batches. These are inspectable code paths, not an end-to-end rerun.

What to trust—and what not to

  • The README reports 91.6% topic placement on a held-out set of 332 questions. That tests agreement with the project’s topic assignments, not correctness on unrelated decisions; I did not reproduce it.
  • The project calls itself a portrait, not a benchmark. Its experiment methods are public, but private result files and the answers cannot be independently checked from this repository alone.
  • Claude wrote synthetic questions, descriptions, and case studies; a local embedding model handles similarity. Jev does not do every step, despite being the only model called by the runtime pipeline.
  • Jev’s own judgments help select and rank experiments. That is an interesting research design and a source of circularity, not independent human validation.

The reusable idea is to preserve each bounded question, its answer distribution, its source, and the transformations used to test it. Typed probabilities make this analysis possible; they do not certify that Jev’s decisions are correct. Repository inspected; pipeline and paid model calls not run.