Nikolay Petrov of Intel explains a 14-minute AI Engineer method for measuring what a SKILL.md teaches a model. The video’s source title says “70%”; the talk treats that as an observation from tested skills, not a universal rate.
Establish that a skill helps
- Turn the work a skill supports into checkable tasks, then run the same tasks with and without the skill.
- His illustrative result moves from three of five tasks solved without the skill to five of five with it. Also compare token cost and time.
- He focuses on knowledge skills that add information, while acknowledging that steering and process skills can be useful without new facts.
Find the useful instructions
- Split a skill into small checkable claims or expected behaviors, each with a question and answer; he suggests roughly 25–30 items per skill.
- Ask the target model questions in two arms, with and without the skill. Use independent judges, avoid questions that leak answers, and choose whether tools are allowed.
- The results form four groups: known either way, newly learned from the skill, harmed by the skill, or unanswered either way.
- Exact parameter names and steps may matter even when a model already knows the general concept; one model’s reported general-versus-exact scores were 70% versus 22%.
Trim only after a task test
- Mark redundant and harmful items as candidates for editing. Petrov reports that harmful items were rare in popular skills he tested.
- Rerun the task evaluations after removing claims the model appears to know already. Keep the change only if performance holds while context cost falls.
- His measurements are examples from an evaluation framework, not evidence that every skill has the same redundancy or that novelty alone predicts task success.
“We can reduce skill size, save tokens, save context.”