Divye Singh builds a Jev-style decision model by training a small scoring layer over a frozen language model. The walkthrough follows the mistakes as closely as the headline score.

This is a separate experimental model. It does not reveal TypeSafe’s proprietary architecture.

Score the options supplied with each question

Singh’s pointer head compares a representation of each option with a final decision representation. The same learned scoring rule handles changing answer lists.

  • The article describes 524,288 trainable parameters in the head; the underlying model stays fixed.
  • Cached representations make repeated head-training experiments cheap, though extracting those representations still has a cost.
  • Empty slots used to batch different-sized answer lists must be masked before computing the training loss.

Singh reports checking the batched loss against separate per-question calculations. An unmasked implementation can return a plausible number while training on padding.

Shuffling answers tests a different property from accuracy

A left-to-right model gives later options access to earlier text. Applying the same scoring rule to every option therefore leaves room for order effects.

Singh shuffles options during training and moves the correct label with them. The final reported order stability is 0.912: the share of evaluated choice and yes/no questions retaining their answer across all tested orders.

A stable answer can still be wrong. Test stability and correctness separately, especially when the correct answer is missing from the supplied list.

Calibrate on validation data, then check a separate test set

Temperature scaling divides scores by a fitted number before turning them into probabilities. It changes confidence while keeping the winning option fixed.

Singh also documents a calibration bug: padded negative-infinity scores produced invalid gradients during temperature fitting. Tests using only finite scores missed it.

For a retrained three-seed experiment, Singh reports 0.963 test accuracy and a 0.055 Brier score, a measure of probability error, after scaling. The earlier notebook’s 0.959 result is a separate single-seed run.

Synthetic tickets establish a pipeline, not production readiness

The labels come from templates with repeated department, tone, and urgency phrases. A high score can reflect those cues. Singh reports 0.882 department accuracy, with the other question types at 0.985 or above in the calibration study.

Typed output is not a correct decision. Real tickets, ambiguous labels, missing-answer cases, and fresh wording need their own held-out evaluation.

Caveat: the complete article was read, but its training and calibration experiments were not run locally; all numerical results above are Singh’s reports.