How to measure whether a skill actually helps with claude plugin eval

This compares results with and without a skill, as scores. claude plugin eval runs the same prompts with your plugin loaded and with no plugin, and shows the difference (Δ). It needs Claude Code v2.1.269 or later, and the skill has to live in a plugin.

Sergii Makarevych's October 19 X article boils it down to two questions: does Claude follow the skill, and does following it make the result better? A grader that checks the skill was invoked answers the first; Δ answers the second.

1. From the plugin root (the folder with plugin.json or .claude-plugin/plugin.json), create the cases. Claude asks what a good result looks like, proposes prompts that should and shouldn't trigger the skill plus graders, and writes them under evals/.

claude plugin eval init

1. Run the suite. Each case runs three times with the plugin and three times without it.

claude plugin eval .

1. Read WITH, W/OUT and Δ in the summary table. A positive Δ means the skill raised the score.

A Δ near zero with a failing tool_used: Skill grader means Claude isn't picking your skill on natural phrasing. Fix the skill's description and run again. While tuning graders, run one case once with no baseline to save cost.

claude plugin eval . --case <case-name> --runs 1 --ablation none

Every run and judge call is a real model call on your account, counted against your plan or API bill. Depending on your account the report is published to claude.ai; add --no-publish to keep it local. To iterate on one skill inside a conversation, use the skill-creator plugin's evals instead. The two tools don't read each other's case files.