Why Claude Isn't Using Your Skill (And How to Fix It)

Skills are lazy-loaded on purpose. That is the entire point of the format — you can ship a 4,000-line reference doc and pay nothing for it until the skill actually runs. But the flip side is the thing that trips everybody up:

• Always in context: the skill's name, its description, and its optional when_to_use.
• Loaded only on invocation: the whole markdown body and every supporting file next to it.

So all those careful instructions, the examples, the "use this when the user asks about X" paragraph you buried in section three — none of it participates in selection. Claude is matching the user's words against one short string, and the combined description plus when_to_use text gets truncated at 1,536 characters in the skill listing to keep context costs down.

That 1,536 characters is your entire budget for winning the match. Spend it accordingly.

In order of how often I hit them:

1. The description describes the skill instead of the request.This is the big one. "Comprehensive deployment tooling for our platform" contains not one phrase a human being would ever type into a prompt. It reads like a package README because that is what you were thinking about when you wrote it.
2. Phrasing drift.Your description says "generate a commit message." Your team says "write me a commit for this." Close enough for a person, not always close enough for a matcher working off one line.
3. It is gated off and you forgot. disable-model-invocation: true pulls the description out of context entirely, so Claude cannot choose the skill on its own — only you can, by name. And a paths glob restricts the skill to files matching that pattern, so it stays invisible everywhere else.

Check number three first. It takes ten seconds and it makes every other fix pointless if you skip it.

The rewrite is mechanical once you accept what the field is for. Stop summarising the skill. Start listing the moments it should fire.

---
name: release-notes
description: Comprehensive release note generation tooling.
---

---
name: release-notes
description: Turns merged PRs since the last tag into release notes.
  Use when the user asks what shipped, wants a changelog, is cutting
  a release, or asks to write up the notes for a version.
when_to_use: |
  "what changed since v2.1", "draft the changelog",
  "we're cutting a release", "write release notes"
---

when_to_use is the underused field here. It exists specifically to hold trigger phrases and example requests, it gets appended to the description in the listing, and it counts toward the same 1,536-character cap. Spend that space on phrasings people actually use, not on more prose about what the skill does. The exact frontmatter rules live in the Claude Code skills reference.

If you are still deciding whether a capability belongs in a skill at all, that is a different question — I went through it in skills vs MCP servers and skills vs subagents.

Here is what everybody does instead. You tweak the description, you open a session, you type a prompt, the skill fires, you declare victory. Then it does not fire for a teammate who phrased it slightly differently, and you are back where you started. One run of a non-deterministic agent tells you almost nothing.

Anthropic shipped the fix for this on 11 September 2026 in v2.1.269. claude plugin eval runs your plugin against a suite of cases, where a case is a realistic prompt plus one or more pass/fail graders, and scores what came back.

claude plugin eval init    # Claude interviews you and writes the cases
claude plugin eval .       # run every case under evals/

init is genuinely good: it reads the plugin, asks you what a good result looks like, proposes prompts that should and should nottrigger it, designs the graders, pilots them once to check they behave, and writes one case directory per prompt. The negative cases matter as much as the positive ones — a skill that fires on everything is its own kind of broken.

Then the part that makes this worth running: each case runs three times with your plugin and three times with no plugin at all. One case is six agent runs. You get two scores and their difference.

CASE        WITH  W/OUT  Δ      RUNS  COST
first-case  1.00  0.33   +0.67  6     $0.41

There are six grader types. Four are free because they are computed from the transcript and the filesystem — regex, tool_used, tool_order, file_exists. Two call a judge model and cost money: llm and baseline.

For "did my skill actually fire," you want tool_used. Drop this in evals/<case>/graders/skill-fired.md:

---
type: tool_used
tool: Skill
input_match: '"skill"\s*:\s*"(?:[\w-]+:)?release-notes"'
---

It passes when Claude invoked that skill at least once in the run, including by its namespaced plugin-name:skill-name form.

One subtlety worth understanding before you misread your first report. A "the skill was invoked" check can never pass in the no-plugin arm — there is no plugin to invoke. Counting it would drag the without-arm toward zero and inflate your delta into a lie. So Claude Code excludes every tool_used: Skill grader from the score in both arms and reports it in the with-arm as a pass/fail indicator only. If you want a grader scored in both arms anyway — which is exactly what a "must not invoke the skill" check with min: 0 and max: 0 needs — set arm: both on it.

Pair it with one grader on the result and one on the path. Result graders on long output should be regex over the produced file, not llm— a judge reading 300 lines gives you a different verdict on Tuesday than it did on Monday.

A high WITH score feels great and means nothing on its own. Three readings, and they point at three different fixes:

• Δ near zero and the Skill grader failing. Claude is not choosing your skill on natural phrasing. Anthropic calls this out as the most common first finding, and it is a description problem. Go back to section three, change the description, re-run.
• Δ near zero and the Skill grader passing, both arms high.The skill fires correctly and the answer is right — and the model would have produced it without you. Uncomfortable, useful, and much better to learn now than after you have asked twelve people to install it.
• Δ negative and the Skill grader passing. Suspect the judge before the plugin. A small judge model will happily mark a correct answer wrong because it is formatted differently from what your rubric described. Re-run with --judge-model sonnet and rewrite the rubric as concrete PASS and FAIL conditions so formatting stops deciding the verdict.

Full grader options and the report format are in the plugin evals documentation.

Every eval run is a real model call on your account. The arithmetic is simple and it compounds fast: cases × runs × 2 arms, plus three judge votes for each llm or baseline grader on every run. A ten-case suite at defaults is sixty agent runs. That is not free, and it is the reason a lot of people set up evals once and never run them again.

Four habits keep it sustainable:

• Iterate on one arm, one run. --case <name> --runs 1 --ablation none while you are editing a description. Then confirm at the default three runs before you believe the improvement.
• Build the every-change suite from free graders only. regex and tool_used cost nothing beyond the agent runs themselves.
• Drop the baseline when you do not need it. --ablation none halves the run. Just remember the absolute score is not comparable to a two-arm run, because nothing gets excluded in single-arm mode.
• Set a ceiling. --max-cost-usd is checked before each run starts. Runs already in flight finish, so you can overshoot slightly.

This is the same discipline that keeps agent token spend from quietly tripling — I wrote about the general version in what a subagent actually costs and cost controls for autopilot agents.

A description that matches today can stop matching when a new model ships. That is the actual argument for putting this in CI rather than running it once after a bug report.

claude plugin eval . \
  --trust-plugin \
  --json results.json \
  --threshold 0.8 \
  --model claude-sonnet-5 \
  --judge-model claude-haiku-4-5 \
  --no-publish \
  --max-cost-usd 20

• --trust-plugin— without it a CI job with no terminal is refused outright with exit 1, or worse, sits waiting at a trust prompt.
• --threshold — defaults to 1.0. Leave it alone and the command exits 1 on any case that is not perfect, which will make you disable the job by week two. Set a bar you actually mean.
• Pin both models— otherwise your scores are not comparable across time and the trend line is meaningless.

Exit codes: 0 means every case met the threshold. 1 means a case scored below it, a case file failed to load, no cases were found, or the directory was not trusted. 2 means a partial run — the cost ceiling was hit or the credential was rejected — and results.json is still written with partial: true. Treat 2 as "we did not measure," not as "the plugin failed." Leave those documents out of any trend you chart.

Before evals existed, a skill that fired felt like a skill that worked. There was no way to separate "my instructions steered the model" from "the model was going to do that anyway." So every skill anyone wrote was, by definition, a success.

The delta ends that. And the honest expectation going in is that some of your skills will come back flat — fires reliably, produces the right answer, contributes nothing the base model was not already doing. That is not a failed skill so much as a skill whose real job turned out to be consistency rather than capability, and it is worth knowing which of those you have before you ask a team to install it or publish it to a private plugin marketplace.

My own rule after wiring this up: if a skill's delta is flat, it either gets deleted or it gets rewritten around the part the model genuinely cannot guess — internal conventions, a specific API contract, the one command that has to run first. That is also the part that survives being moved between agents, which is why a portable SKILL.md and a high-delta SKILL.md tend to be the same file.