Hamel Husain and a co-host spend an hour live-testing claude plugin eval — the plugin-evaluation feature Anthropic shipped for Claude Code (announced from the Claude Devs X account) — by pointing it at a plain-writing/tone skill. They never get a useful eval out of it, and the diagnosis of why is the interesting part.

What the tool claims to do

  • Run claude plugin eval from the root of a plugin folder to “see what value your plugin is adding or if it needs more work”
  • The advertised loop: draft test cases → run the plugin against them → score the runs → re-run every case without the plugin and compare
  • Bring a few real prompts, tell it what good and bad output looks like, and it pilots the suite, estimates the cost of a full run, and writes per-case scores plus an HTML report

What actually happened

  • Small print, discovered live: it only evals plugins, not skills. The skill they wanted to test sat in a repo whose evals/ directory was already occupied by a different harness, so the tool refused to touch it as-is
  • First contact was opaque — it started printing scores before they had supplied examples or criteria: “we haven’t given it any examples or told us what it’s looking for… it just started printing out a bunch of scores”
  • It eventually produced a real artifact: three graders, a JSONL dataset, aggregate_results.json, and an HTML report (“slop eval dashboard”) with criteria like “no em dash” and “skills fired”, each with a weight and a pass/fail mark

The critique: the eval isn’t actionable

  • No provenance. Every number, criterion and skill appears with no indication of where it came from — “I want to be able to look at them and know where they came from and this doesn’t tell us at all”
  • It’s a single ablation of the skill against itself. Baseline = no skill; treatment = skill. The skill of course satisfies the criteria derived from the skill’s own rule list, which measures compliance, not quality
  • The labels are vibey. Auto-generated graders are regex- and judge-shaped guesses, and the docs’ escape hatch — writing the graders yourself — puts you back to doing the work by hand
  • Their name for the output: “vibe evals” — “it’s like vibe coding… we don’t really know anything, let’s just look at more things and see if we can generate some stuff”

The process gap: it assumes the hard part is already done

  • The tool asks for real failure modes from your own use of the skill — which is the output of error analysis, not something you can recall on demand: “there’s no way you can recall these things from memory, you have to do some error analysis first”
  • Nothing in the flow pushes you to build an annotation dataset or look at traces; it assumes you arrive with one
  • Their summary: “it’s mostly asking you for all the stuff that’s a lot of the hard parts of the eval process anyway — and if you have that, you’re almost all the way to evals”
  • Timing problem. Feedback has to be captured in situ, while you’re actually using the skill. A chat prompt half an hour later can’t reconstruct what annoyed you
  • Coverage is arbitrary: the skill has ~25 concrete rules, the tool produced about five tests. Asking for 25 matching cases made it write a generator script for synthetic data instead of mining real examples

The one genuinely useful piece

  • Fire / should-not-fire tests. The concrete win was being able to check that the skill fires when it should and — more importantly — doesn’t over-fire
  • Testing trigger behaviour is a real failure mode, and it’s the part both hosts said they’d keep

Rough edges seen live

  • Generated regex graders that threw YAML parse errors — “I already found a bug. Several regex graders… throwing YAML parse errors”
  • Long opaque stretches: the pilot kept “cranking” with a shell running in the background and no explanation of what it was doing, while continuing to ask for feedback
  • The interface keeps you in the loop without telling you what’s going on — the opposite of what you want from an eval runner

What they’d build instead

  • An eval mode you turn on while working, with an annotation side panel that collects feedback the moment the skill fires
  • Capture multiple states: the AI’s output, then the same file minutes later — so you can diff what the human changed by hand, and even see whether a sentence survived into the final published piece
  • Somewhere in the loop a browser-based annotation UI, hosted by the coding agent, rather than a chat transcript
  • The bar they set for the tool itself: guide people through choosing examples and looking at data, which is the part the current flow skips

Verdict

  • “I think we just call this a failure” — a red FAIL stamp, with the concession that it’s a first pass and may suit less ambiguous work like coding better than prose
  • Worth watching whether Anthropic’s next revision inverts the loop: collect feedback while the skill runs, then propose tests from it

“It’s mostly kind of asking you for all the stuff that’s a lot of the hard parts of the eval process anyway. And it’s like, well, if you have that, you’re almost all the way to evals.”