LLM Classification Is Feature Engineering — Taylor

Asking an LLM for a label sucks to work with — and the annoying part is that it often performs decently. Taylor, a statistician who now works on generative AI at YouTube, argues the problem is framing: you are using a feature extractor as a decision maker.

Measured against the things you actually want from a classifier, the LLM verdict comes up short:

• Calibration. Verdicts are hard labels. Token log probabilities exist but nothing supports treating them as calibrated, and asking the model for its confidence gives you a number with no better claim. Without probabilities you cannot trade precision against recall in any principled way.
• Using all your information. You can paste structured fields into the prompt, but the LLM has no obligation to use them — and no way for you to check. Its baked-in prior is also a bad fit for your population: it does not know whether the positive class is rare or enriched, and telling it means editing the prompt per population with no guarantee it lands.
• Interpretability. A prompt reads as transparent prose, but you have no idea which parts are being followed and which are being ignored.

None of that is a fault of the model. It was not designed as a classifier.

The fix is one line

Wrap the verdict in a logistic regression:

p(y=1|x) = σ(α + β · LLM(x))

The trick is noticing the special case. Set β to infinity and you recover the LLM classifier exactly — which means the current state of the art is one particular, and frankly careless, choice of parameter. Estimate α and β from training data instead and you get two empirical numbers per verdict. In Taylor’s irony example, P(ironic | verdict “Ironic”) comes out at 0.687 and P(ironic | verdict “Not”) at 0.188.

Suddenly the desiderata are all satisfiable. You get probabilities calibrated in expectation, an operating threshold you can move deliberately, a place to put covariates, and the ability to reweight examples toward a different population. Interpretability improves too — not of the verdict itself, but of how much it contributes to the final decision.

The bigger payoff is the upgrade path

This is the part that matters more than the calibration. When an LLM-as-classifier underperforms, your only move is to fiddle with the prompt — “an arcane undertaking about which advice abounds on the internet but wisdom is scarce.” The feature-engineering framing hands you the standard ML levers:

• Collect more data. The honest cost: you lose the training-free appeal. The counter is that you were going to need a test set anyway.
• Improve the features. If a feature should always imply a positive, check that empirically and debug when it does not. You can also score the features themselves as a secondary target and recurse.
• Add features. The log probability of the verdict token, multiple runs of the classifier (with reasoning before the verdict, those log probabilities saturate at 0 and 1), or more subquestions from the model.
• Change the model. Logistic regression is a starting point, not a religion. Gradient boosting, a neural net, or a rules engine all plug in.

The experiment

SemEval 2018 Task 3 — irony detection in tweets. 4,618 tweets from expert annotators, 3,834 for training and 784 for testing, run through a flash-tier Gemini model at temperature 0 with a JSON schema for the output.

The prompt alone, with no training of any kind, gets F1 0.747 and a Brier score of 0.259. Since 0.25 is what random guessing scores, that is a striking split: genuinely useful accuracy paired with useless probabilities.

The regression alone does not move F1, which is expected — it is monotone, so it cannot reorder anything. What it fixes is the probability. The 19 LLM features are boolean subquestions (is it humorous, is it a complaint, is there a contrast, feigned surprise, mock enthusiasm, puns, backhanded compliments), and the 9 rule-based ones are computed directly (is it a reply, does it contain a URL, character length, counts of hashtags and exclamations, all-caps words, emoji). The deterministic features lie entirely outside the model and still add a point of F1.

For context, on the same test set: random baseline 0.373, the paper’s SVM + tf-idf baseline 0.589, the 2018 competition winner 0.705, and a post-competition LSTM with attention 0.786. The bare prompt beats the competition winner. Logistic regression on LLM-extracted features reaches 0.779 with confidence intervals overlapping the post-competition state of the art.

The workflow worth stealing

The most useful part is not the result — it is how the features got written. Taylor dumps the ten misclassifications with the model’s reasoning attached and reads them. The errors are what produced prompts 5, 13, 14 and 15 in the expanded feature set: sentiment mismatch, endorsement of a bad outcome, feigned surprise, mock enthusiasm. That is residuals driving feature design, which beats tweaking adjectives in a prompt and re-rolling.

Caveats

• The headline comparison is F1-only, and threshold choice interacts with F1 in ways an overlapping confidence interval does not settle. The calibration improvement is the more defensible claim.
• Nineteen boolean features over 3,834 training rows is a lot of freedom. Regularization is not discussed, and the last point of F1 (0.768 to 0.779) comes from 784 test rows, so it is close to noise.
• Irony is an unusually kind task for this approach, because worldliness is exactly what the LLM brings. Where the LLM’s prior is the problem rather than the asset, this framing helps less.

It also pairs neatly with the argument that you should not ask an LLM to emit a label from a large taxonomy at all, but to invent one and resolve it by embedding — the same underlying move, stated from the other end. In both cases the model’s output is a feature, and the decision belongs somewhere cheaper and more measurable.

The author’s own next step is agentic classifiers: let the model investigate, then use features of the investigation — how rigorous and comprehensive it graded its own work — as covariates in the final model.