skill-eval-builder

Compare original and translation side by side

🇺🇸

Original

English

🇨🇳

Translation

Chinese

Skill Eval Builder

Skill 评估构建器

Overview

概述

Set up a small, real eval for a skill: an

evals/

folder next to it with a few real cases, a runnable script, and a scorecard. It measures what scripts can't pin down — does the skill fire when it should (and stay quiet when it shouldn't), is its output valid, is it within a time budget.

Core principle: an eval is a folder, not a framework. Keep it that small.

为某个Skill搭建小型真实评估：在Skill目录旁创建一个

evals/

文件夹，包含若干真实测试用例、可运行脚本和评分卡。它能测量脚本无法确定的内容——Skill是否在应当触发时触发（且在不应触发时保持静默）、输出是否有效、是否在时间预算内完成。

核心原则： 评估是一个文件夹，而非框架。保持轻量化。

Inputs

输入

Target skill — path or name (the dir with its
```
SKILL.md
```
).
A few real cases — should-fire and shouldn't-fire prompts + real inputs; help the user find them if needed.
Which dimensions matter — default invocation + validation; add duration/others if they fit.

目标Skill —— 路径或名称（包含
```
SKILL.md
```
的目录）。
若干真实用例 —— 应当触发和不应触发的提示词+真实输入；必要时协助用户寻找这些用例。
关键评估维度 —— 默认包含触发验证+输出验证；若适用可添加时长等其他维度。

Steps

步骤

Read the target skill — its
```
SKILL.md
```
, references, and scripts, so you know what it does and what it produces.
Pick dimensions that fit it, from
```
references/eval-dimensions.md
```
. Workflow skill → invocation + duration; capability skill → validation-heavy.
Gather 3–5 real cases — should-fire and shouldn't-fire prompts plus real inputs. Always include at least one quiet case.
Define "good" per case — a
```
fired_when
```
side-effect check, a
```
validate
```
command, and a
```
budget_s
```
from a real baseline (see
```
references/eval-anatomy.md
```
).

Scaffold the artifact next to the skill:

bash

python3 "${CLAUDE_SKILL_DIR}/scripts/scaffold-evals.py" <target-skill-dir>

Then fill in

evals/cases.md

with the cases from steps 3–4.

Check, then run the baseline:

bash

python3 <target-skill-dir>/evals/run.py --dry-run   # cases parse?
python3 <target-skill-dir>/evals/run.py             # real run (calls claude -p)

Report the scorecard and where it lives.

读取目标Skill —— 查看其
```
SKILL.md
```
、参考文档和脚本，明确其功能与输出内容。
选择适配的评估维度，参考
```
references/eval-dimensions.md
```
。工作流类Skill → 触发验证+时长；能力类Skill → 侧重输出验证。
收集3–5个真实用例 —— 包含应当触发和不应触发的提示词及真实输入。务必至少包含一个静默用例。
为每个用例定义“合格标准” —— 包括
```
fired_when
```
副作用检查、
```
validate
```
命令，以及基于真实基准的
```
budget_s
```
（详见
```
references/eval-anatomy.md
```
）。

在Skill旁生成评估工件：

bash

python3 "${CLAUDE_SKILL_DIR}/scripts/scaffold-evals.py" <target-skill-dir>

随后将步骤3–4中的用例填入

evals/cases.md

。

检查并运行基准测试：

bash

python3 <target-skill-dir>/evals/run.py --dry-run   # 验证用例是否可解析？
python3 <target-skill-dir>/evals/run.py             # 真实运行（调用claude -p）

报告评分卡结果及其存储路径。

Testing a classification gate

分类网关测试

If the skill opens with a classifier (routes feature vs bug, or decides whether to continue), test that gate:

python3 "${CLAUDE_SKILL_DIR}/scripts/scaffold-evals.py" <target> --classify

Fill
```
evals/classify-cases.md
```
with the labels, the gate's own instruction, and real labeled examples (e.g. the last 10 tracker tickets + their existing labels as ground truth). See
```
references/eval-anatomy.md
```
.
```
python3 <target>/evals/classify.py
```
→ an accuracy + confusion scorecard.

若Skill以分类器开头（区分功能与Bug，或决定是否继续执行），需测试该网关：

python3 "${CLAUDE_SKILL_DIR}/scripts/scaffold-evals.py" <target> --classify

在
```
evals/classify-cases.md
```
中填入标签、网关自身指令，以及真实标注示例（例如最近10个跟踪工单及其现有标注作为基准）。详见
```
references/eval-anatomy.md
```
。
```
python3 <target>/evals/classify.py
```
→ 生成准确率+混淆矩阵评分卡。

Output format

输出格式

The scorecard (dimensions × cases) plus the saved artifact path:

┌───────────────────────────────┬──────────┬────────┬────────────────────┐
│ Case                          │ Fires?   │ Valid? │ Duration vs budget │
├───────────────────────────────┼──────────┼────────┼────────────────────┤
│ "add tests for X"             │ ✅       │ ✅     │ 40s / 60s ✅       │
│ "refactor Y" (shouldn't fire) │ ✅ quiet │ –      │ –                  │
└───────────────────────────────┴──────────┴────────┴────────────────────┘

Saved to

<target-skill>/evals/

(cases.md, run.py, results.md).

评分卡（维度×用例）加上保存的工件路径：

┌───────────────────────────────┬──────────┬────────┬────────────────────┐
│ Case                          │ Fires?   │ Valid? │ Duration vs budget │
├───────────────────────────────┼──────────┼────────┼────────────────────┤
│ "add tests for X"             │ ✅       │ ✅     │ 40s / 60s ✅       │
│ "refactor Y" (shouldn't fire) │ ✅ quiet │ –      │ –                  │
└───────────────────────────────┴──────────┴────────┴────────────────────┘

保存至

<target-skill>/evals/

（包含cases.md、run.py、results.md）。

Guidelines

指南

Prefer an observable side-effect for firing (a file the skill produces) over grepping prose.
Runs are not byte-identical — a skill eval runs a model; that variance is what you measure. Don't fake determinism, and don't eval what a deterministic script already guarantees (unit-check the script instead).
Keep it to 3–5 real cases with at least one quiet case. See
```
references/eval-anatomy.md
```
.

优先使用可观测的副作用（如Skill生成的文件）来判断是否触发，而非搜索文本内容。
运行结果并非字节级完全一致——Skill评估会调用模型；这种差异正是需要测量的内容。不要伪造确定性，也不要评估确定性脚本已能保证的内容（这类内容应通过单元测试验证）。
保持3–5个真实用例，且至少包含一个静默用例。详见
```
references/eval-anatomy.md
```
。

Files

文件说明

scripts/scaffold-evals.py

— creates

evals/{cases.md, run.py, results.md}

, or the

classify-*

set with

--classify

```
scripts/run-evals.py
```
— runs cases via
```
claude -p
```
, times each, checks firing + validation, prints/saves the scorecard.
```
--dry-run
```
parses without spending tokens.
```
scripts/classify-evals.py
```
— classifies labeled examples via
```
claude -p
```
, reports accuracy + confusion.
```
--dry-run
```
parses without tokens.
```
references/eval-dimensions.md
```
— the dimension menu + when each applies.
```
references/eval-anatomy.md
```
— cases format, the side-effect pattern, anti-patterns.

If a script can't run here (needs

python3

and the

claude

CLI, or a different OS): don't abandon the task — an eval is just a

cases.md

plus a small runner, so run the cases and tally the scorecard with whatever tools are available.

scripts/scaffold-evals.py

—— 创建

evals/{cases.md, run.py, results.md}

，添加

--classify

参数则生成

classify-*

系列文件。

```
scripts/run-evals.py
```
—— 通过
```
claude -p
```
运行用例，记录每个用例的时长，检查触发情况与输出有效性，打印并保存评分卡。
```
--dry-run
```
参数仅解析用例，不消耗token。
```
scripts/classify-evals.py
```
—— 通过
```
claude -p
```
对标注示例进行分类，报告准确率+混淆矩阵。
```
--dry-run
```
参数仅解析用例，不消耗token。
```
references/eval-dimensions.md
```
—— 评估维度选项及适用场景说明。
```
references/eval-anatomy.md
```
—— 用例格式、副作用模式及反模式说明。

若脚本无法在此运行（需要

python3

和

claude

CLI，或不同操作系统）：不要放弃任务——评估本质上就是

cases.md

加一个小型运行器，因此可以使用任何可用工具运行用例并统计评分卡结果。