Kanwal Mehreen trains a local intent classifier using Unsloth’s decision-model tools and Qwen3.5-4B. The tutorial shows how to prepare labeled examples, fine-tune a model and calibrate its probabilities. It does not test Jev itself.
Start with a small, fixed choice set
- The author selects 10 intent categories from CLINC150 and uses its original splits: 1,000 training, 200 validation and 300 test examples.
- Each training record contains the message as state, the available Choice question and the correct label.
- The walkthrough uses 4-bit loading and low-rank adaptation, which trains a small set of added parameters rather than the whole base model, plus a decision head.
- The reported run uses a Colab Tesla T4 and 60 training steps. Calibration uses the validation set; the held-out test set is evaluated afterward.
Treat the perfect score as a narrow result
The author reports 18.5% validation accuracy before training and 100% afterward, then 100% on the 300 selected in-scope test examples. The author also reports zero test expected calibration error and loss. None of these results was independently reproduced here.
The experiment excludes out-of-scope examples and uses only ten of the dataset’s 150 intents. It does not establish performance on unfamiliar categories, ambiguous inputs or live traffic. A valid typed answer can still be wrong, and confidence needs separate testing on representative data.
Verification: read the full tutorial and its displayed code and outputs. The training, calibration and metrics were not rerun; the article does not provide a Jev comparison.