Stephen Solka wanted to separate political fundraising and advocacy from ordinary email. He wrote the labeling policy first, tested Jev on real mail, then asked a harder question: could a small, private CPU classifier do enough of the job? I read the full experiment and code examples but did not run the models or reproduce his measurements.

A clean pilot is not an inbox-wide guarantee

  • The author froze a balanced pilot of 100 real emails—half political, half not—and reports that Jev got 99/100 right across seven bulk requests.
  • He then mined boundary cases, including humanitarian relief sent by a campaign and advocacy hidden in otherwise ordinary messages. On a deliberately difficult reserved set, Jev got 19/23. This selected challenge set does not estimate general inbox error rates.
  • The labels were assistant-reviewed rather than human-adjudicated; the original private mail and its annotations cannot be independently checked.

Local alternatives and a smaller model

  • On a separate public synthetic set of 112 emails, Solka reports Jev and Winnow 12B at 109/112 each. Their latency figures use different units: Jev’s bulk request versus the open model’s single email.
  • Confidence filtering can keep the most certain predictions and leave the rest for review. The cutoffs were chosen after inspecting this evaluation set, so their observed accuracy and coverage are not validated deployment promises.
  • A SetFit classifier built from a compact embedding model and 120 labeled private examples reached a reported 98/100 on the original pilot but 18/23 on the hard challenge.
  • Exporting the entire pipeline to ONNX and quantizing produced a reported 23.4 MB model file for CPU inference. One label changed at a larger batch size in the author’s quantization audit: an unchanged model architecture does not imply unchanged decisions or confidence.

The action still needs a safety policy

The sample routing code sends only sufficiently confident political predictions to Trash, leaving uncertain messages in the inbox. The cutoff needs separate validation at the actual batch size and input length; a truncated email can hide the very appeal the classifier is meant to catch. The post describes a prospective adapter, not a verified live mailbox deployment.

The useful pattern is policy → frozen pilot → error mining → separate challenge → smaller local model → measured routing threshold. Typed probabilities are not proof that either the classification or the automated move is right.