Nicia tests whether an agent’s proposed knowledge-base write can land automatically or needs a person to review it. The repository includes labeled synthetic cases, a runnable multi-provider harness and per-case results. Its November 11 run compares nine hosted decision-model backends on the same task.

What the gate checks

  • Each request compares an incoming record with accepted records for contradiction or restatement. An edit also gets checked against its prior version.
  • The code admits a write only when the highest contradiction and restatement probabilities are both below 0.2. Ambiguous answers and failed calls go to review.
  • Nicia’s test has 85 synthetic cases per pass: 56 require review and 29 are routine. The seven passes per backend also reverse record order and vary concurrency.

What Nicia measured

  • Jev 1.14.0, Clef and Microsoft Decision 1 meet Nicia’s recall, routine-admission, latency and zero-failure gates in all seven passes. Passing still allows a wrong admit: Jev misses one review-required restatement in both selected orders. Clef misses two review-required cases in the selected original order; Microsoft misses one with records reversed.
  • Clef-flash admits only 19 of 29 routine writes in the selected original-order pass and 18 of 29 in reverse. It meets all gates in zero of seven passes.
  • Under the same Clef-flash model name, stored evidence changed on every case versus October 31. Nicia reports changed decisions too, but cannot identify whether weights, serving or another endpoint behavior caused it.
  • The published reports contain 5,355 scored calls. Those are repeated calls on the same cases, not 5,355 independent examples. The hosted models used different provider paths, so caller-observed latency is not a pure inference-speed ranking.

The script and saved reports make this a useful workload-specific comparison. I inspected the implementation and results at revision 36b53a2bae3764fb32bc1ad5eb8e2a60dce11af0 and ran its eight credential-free adapter tests; I did not rerun hosted inference. Caveat: thresholds were tuned on Jev, and the evidence maximum is not a calibrated chance that an admitted write is correct. Typed output still needs a review policy and a decision-quality check on real writes.