Miles Roxas’s Skill Studio runs agent skills on test cases, grades their outputs and lets people review them blind. Its Jev integration covers rule adherence, writing, code review, workflow traces and design artifacts. Runs, judge answers and human ratings stay in files beside the skill.

Separate the judgment from the scoring rule

The application sends narrow questions through TypeSafe’s SDK, then stores the returned probabilities and verdicts in report.json. Local scoring code turns those answers into layer scores and a weighted total.

  • Change weights and thresholds without asking Jev the same questions again.
  • Keep the model identifier and input-token count with each report.
  • Compare saved skill versions and runs with or without the skill.
  • Use code checks alongside model judgments, with each layer visible separately.

The writing pack checks claims against supplied evidence passages. A “contradicted” answer triggers another, more literal conflict question before scoring. That adds a check within the same model; it supplies no independent verification of the claim.

Review the judge before trusting its grade

The blind-review screen hides the writer and scores until a person saves a rating. Its agreement view separates cases where both sides approve, both reject, or the judge misses a human objection.

The review queue prioritizes uncertain judgments and cases with widely varying sample scores. That helps direct attention, but agreement on this selected queue needs a separate representative sample before generalizing.

“Anchors” are saved outputs labeled good or bad. The self-test checks whether each bad anchor scores below its case’s good anchor on the selected layers.

Missing checks change what the total means

Two implementation details deserve attention when using the results:

  • A failed judge pack becomes null and receives a failure flag. The total rescales over the remaining scored layers.
  • The anchor self-test skips a bad anchor when its matching good anchor is missing. The command can report success with no comparisons.

A high total can therefore cover fewer checks than another run. Read the failure flags and layer coverage, and confirm that every intended anchor comparison actually happened.

Typed answers preserve the structure of a judgment. Correct grades still require task-specific human labels and tests of the scoring rules.

Caveat: this is an inspected implementation, not a measured demonstration of judge accuracy. The app executes agent-written code; its security documentation recommends a container or virtual machine for untrusted skills.

Verification: inspected the README, grading documentation, Jev client, judge packs, scoring, anchor self-test and review code at commit 7864c3f. No application run, live Jev calls or performance reproduction was performed; this was not a security audit.