Hamel Husain and Antaripa Saha compare decision models as automated reviewers of an apartment-leasing assistant. Their study scores 98 production traces, logs of conversations and tool actions, against 61 known failures and 37 other traces. Two internal test examples are excluded from performance results.

Choose the error tradeoff first

A threshold turns a failure score into an alert. Lowering it usually catches more failures and raises more false alarms.

  • Precision measures how many alerts correspond to known failures.
  • Recall measures how many known failures the reviewer catches.
  • The authors tune thresholds on training folds and measure predictions on separate examples, using the same splits across models.
  • Their preferred candidates must reach both precision and recall minimums during threshold selection.

A default cutoff can hide a model’s ability to rank examples. Tuning still needs a separate check on unseen cases.

Jev’s calibration and its failure detection tell different stories

The authors find Jev’s probabilities already fairly well calibrated on this dataset: groups with similar scores have roughly corresponding failure rates. Their probability adjustment barely changes Jev’s Brier score, a measure of squared probability error.

Jev nevertheless falls outside their preferred candidate group for the chosen precision–recall requirements. Its saved held-out summaries report roughly 71% precision and 87% recall after threshold tuning. Good calibration can coexist with too many false alarms. Typed output is not a correct decision.

Read the cost chart alongside the fallback record

The study reports F1 scores of 0.79 for Perplexity and 0.87 for OpenAI Decisions, at 0.82 and 3.72 cents per 100 evaluation calls, respectively. F1 combines precision and recall. Those are the authors’ API-cost calculations and evaluation results; live inference was not reproduced here.

The threshold audit matters as much as the chart:

  • Perplexity could not meet both training minimums in 36 of 100 splits; Liquid d1-ultra failed in 29.
  • Those splits fall back to unconstrained F1 tuning and remain in the reported validation averages.
  • The training minimums therefore promise no equivalent floor on future examples.

The authors recommend narrower reviewers for individual failure types and measuring the whole workload before choosing a cheaper model.

Verification: read the full article and checked its published score and threshold JSON. Recomputing default-threshold F1 matched the saved values for all six models; fallback counts also matched. Original conversations are private, and neither human labels nor provider performance were independently reproduced.