Chris (kqr) rebuilt his LLM code-comment classifier on public data and a sturdier foundation, then wrote up the parts that were actually hard. The classifier — logistic regression over stylometric and part-of-speech features — is the least interesting component. The dataset work and the deployment constraint are the story.

The numbers, with the caveat attached:

  • Balanced accuracy 77% under cross-validation, rising to 88% on a smaller non-synthetic set of real human/robot comments.
  • Aggregate error rate is ~25% either way, which sounds bad — but predictions are calibrated, so at ≥80% confidence the false-positive risk drops to about 5%.
  • The verdict’s own percentage is the useful signal; the confusion matrix is mostly there because readers expect it.

The dataset recipe is where the cost landed. Sample pre-2021 commits from permissive/copyleft repos for human comments, strip the comments, have LLMs regenerate them — and balance token counts per file across classes so the model can’t learn which repository instead of which writing style. Mistakes that forced full recollections (twice, at real expense):

  • Different source files per LLM to comment on — subject-matter leakage.
  • Generating comments for files with almost no human comments.
  • Docstrings left in place, letting models imitate repository style.
  • Missing comment syntaxes across languages.
  • Keeping trivial human comments like “chIcon” or “Alias”, which teaches the classifier that humans write like shit.
  • One fixed generation prompt, which narrowed the variation the model needs to learn from.

Feature selection was guided by single-feature discriminative power across several pairwise tasks rather than brute-force subset search (15 features would mean 30,000+ classifiers, five folds each). POS-tag n-grams won regardless of tagger; character n-grams were strong; raw word frequencies are powerful but dangerous because they encode subject matter. The model is trained 7-way, has probability mass from two overlapping models smeared back out, and is renormalized to a 50/50 human/robot prior — with a temperature coefficient scaled by the square root of input length to fix L1 normalization’s tiny probabilities.

Then shipping drove architecture. The raw model was several megabytes; quantization plus a document-frequency filter (knee at ~0.05%) brought it to 355 kB so it could run client-side. Browser-side POS tagging ruled out spaCy and nltk and forced wink-nlp — so the Python training pipeline shells out to Node to keep train and inference in the same engine.

The takeaway: a ~77% classifier with honest per-prediction confidence is more useful than a headline accuracy number, and “the data was junk” was discovered twice, late, at cost. Also worth remembering before wiring any such detector into CI — it’s a writing-style classifier for code comments, and the author makes no promises outside that domain.