skill-evaluator

Compare original and translation side by side

🇺🇸

Original

English

🇨🇳

Translation

Chinese

Skill Evaluator

Test whether a skill is correct, discoverable, and valuable — not just whether its unit tests pass. The harness is agent-agnostic: it drives whichever coding-agent CLI the user uses, because Agent Skills are portable across all of them.

测试某个skill是否正确、可被发现且具备价值——而不仅仅是其单元测试是否通过。该测试框架与Agent无关：它可以驱动用户使用的任意编码Agent CLI，因为Agent Skill可在所有这些工具间移植。

When to use which layer

各层级的适用场景

Three layers, increasing cost and fidelity (full rationale in

references/methodology.md

Layer	Question	Script	Needs a CLI?
1. Deterministic	Do the scripts emit the documented numbers?	`run_script_checks.py`	No
2. Trigger	Does the description activate on the right prompts?	`run_trigger_eval.py`	Yes
3. Quality	Does following the SKILL.md beat no skill?	`run_quality_eval.py` → grade → `aggregate_benchmark.py`	Yes

Always run Layer 1 (it's free). Add Layers 2–3 when you can run a coding-agent CLI.

分为三个层级，成本和保真度依次提升（完整原理见

references/methodology.md

）：

层级	待解决问题	脚本	是否需要CLI？
1. 确定性检查	脚本是否输出文档中记录的数值？	`run_script_checks.py`	否
2. 触发测试	描述信息是否会在正确的提示词下激活？	`run_trigger_eval.py`	是
3. 质量评估	遵循SKILL.md是否优于无Skill的情况？	`run_quality_eval.py` → 评分 → `aggregate_benchmark.py`	是

务必运行第1层（无需成本）。当可以运行编码Agent CLI时，再添加第2-3层的测试。

Step 0 — pick the agent CLI

步骤0 — 选择Agent CLI

Ask the user which coding agent they use, then map it to an adapter id. Supported:

claude-code

openai-codex

antigravity

(the

agy

CLI that replaced Gemini CLI on 2026-06-18),

cursor-cli

github-copilot-cli

amp

opencode

grok-cli

. See the full matrix and auth in

references/adapters.md

, or run:

bash

python scripts/agent_adapters.py list

Confirm the binary is installed and the auth env var is set (the matrix lists it). Before any real run, dry-run it to see the exact command:

bash

python scripts/agent_adapters.py build <agent> --prompt "test" --workdir /tmp/wd

询问用户使用的编码Agent，然后将其映射到对应的适配器ID。支持的Agent包括：

claude-code

、

openai-codex

、

antigravity

（2026年6月18日取代Gemini CLI的

agy

CLI）、

cursor-cli

、

github-copilot-cli

、

amp

、

opencode

、

grok-cli

。详见

references/adapters.md

中的完整矩阵及认证信息，或运行以下命令查看：

bash

python scripts/agent_adapters.py list

确认对应的二进制文件已安装，且已设置认证环境变量（矩阵中列出了所需变量）。在正式运行前，先执行试运行以查看具体命令：

bash

python scripts/agent_adapters.py build <agent> --prompt "test" --workdir /tmp/wd

Step 1 — deterministic script checks (always)

步骤1 — 确定性脚本检查（必选）

bash

python scripts/run_script_checks.py --skill <path-to-skill> --json

Runs the

script_checks

in the skill's

evals/evals.json

, executing each script and grading its

--json

output against machine-checkable assertions. Exit non-zero on any failure — safe for CI. If the skill has few/no

script_checks

, add them for every eval whose answer is computable (schema in

references/schemas.md

); this is the cheapest, most durable guard against doc↔code drift.

bash

python scripts/run_script_checks.py --skill <path-to-skill> --json

运行skill的

evals/evals.json

中的

script_checks

，执行每个脚本并将其

--json

输出与可机器校验的断言进行对比。若有任何失败则返回非零退出码——适用于CI流程。如果skill的

script_checks

很少或没有，请为每个结果可计算的评估项添加检查（格式见

references/schemas.md

）；这是防范文档与代码不一致的最经济、最可靠的手段。

Step 2 — trigger / discovery eval

步骤2 — 触发/发现评估

Does the description fire on the right prompts and stay quiet on near-misses?

bash

undefined

描述信息是否会在正确的提示词下触发，而在近似但不相关的提示词下保持静默？

bash

undefined

Dry-run first (prints the per-CLI commands, runs nothing):

先试运行（打印每个CLI的命令，不实际运行）：

python scripts/run_trigger_eval.py --skill <path> --agent <agent> --dry-run

Real run with a labelled query set (~20: half should-trigger, half near-miss):

使用带标签的查询集进行正式运行（约20个：一半应触发，一半为近似但不相关的案例）：

python scripts/run_trigger_eval.py --skill <path> --agent <agent>
--queries queries.json --runs-per-query 3 --json


Design the query set per `references/methodology.md` (positives + tricky
negatives). Without `--queries`, the skill's eval prompts are used as
should-trigger cases — add negatives for a real discrimination test.

python scripts/run_trigger_eval.py --skill <path> --agent <agent>
--queries queries.json --runs-per-query 3 --json


根据`references/methodology.md`设计查询集（包含正例和有挑战性的反例）。若未指定`--queries`，则会使用skill的评估提示词作为应触发的案例——添加反例才能进行真正的判别测试。

Step 3 — output-quality eval (the with/without delta)

步骤3 — 输出质量评估（有无Skill的差值对比）

The headline measure: does an agent following the SKILL.md beat no skill?

bash

undefined

核心衡量指标：遵循SKILL.md的Agent是否优于无Skill的情况？

bash

undefined

1. Dry-run the plan (no tokens spent):

1. 试运行计划（不消耗令牌）：

python scripts/run_quality_eval.py --skill <path> --agent <agent>
--workspace <skill>-workspace --dry-run

2. Real run: with-skill AND no-skill baseline, isolated clean dirs each:

2. 正式运行：同时执行带Skill和无Skill的基准测试，各自使用独立的干净目录：

python scripts/run_quality_eval.py --skill <path> --agent <agent>
--workspace <skill>-workspace --iteration 1 --json


This installs the skill into a temp project skills dir for the with-skill run,
runs a clean baseline without it, and captures `outputs/`, `response.txt`, and
`timing.json` per run.

**Then grade each run** against its `assertions` and write `grading.json`
(`references/grader.md` — re-derive numbers, require concrete evidence, no partial
credit, critique weak assertions). For mechanically checkable assertions, reuse
Layer 1 rather than eyeballing.

**Then aggregate** into the benchmark with the delta:

```bash
python scripts/aggregate_benchmark.py <skill>-workspace/iteration-1 \
  --skill-name <name> --agent <agent> --json

run_summary.delta.pass_rate

is the value of the skill. Surface patterns the averages hide (

references/methodology.md

): non-discriminating assertions, high-variance evals, time/token tradeoffs.

Then generate the review and put it in front of the user before you self-grade (a standalone HTML page — no server needed):

bash

python eval-viewer/generate_review.py <skill>-workspace/iteration-1/benchmark.json -o review.html

It renders the with/without delta, per-configuration stats, and an expandable per-eval breakdown of each graded assertion (text, pass/fail, evidence).

python scripts/run_quality_eval.py --skill <path> --agent <agent>
--workspace <skill>-workspace --iteration 1 --json


该步骤会将skill安装到临时项目的skills目录中用于带Skill的运行，同时在无Skill的情况下运行干净的基准测试，并为每次运行捕获`outputs/`、`response.txt`和`timing.json`。

**然后针对每个运行结果进行评分**，对照其`assertions`并编写`grading.json`（见`references/grader.md`——重新推导数值，要求提供具体证据，不给予部分分数，对薄弱的断言提出批评）。对于可机械校验的断言，重用第1层的检查而非人工检查。

**然后将结果汇总到基准测试中并计算差值**：

```bash
python scripts/aggregate_benchmark.py <skill>-workspace/iteration-1 \
  --skill-name <name> --agent <agent> --json

run_summary.delta.pass_rate

即为该skill的价值所在。要关注平均值背后的模式（见

references/methodology.md

）：无判别性的断言、高方差的评估、时间/令牌的权衡。

然后生成评估报告，并在自我评分前展示给用户（一个独立的HTML页面——无需服务器）：

bash

python eval-viewer/generate_review.py <skill>-workspace/iteration-1/benchmark.json -o review.html

该页面会渲染有无Skill的差值、各配置的统计数据，以及每个已评分断言的可展开详细 breakdown（文本、通过/失败、证据）。

Step 4 — iterate

步骤4 — 迭代优化

Improve the skill from the signals (failed assertions, weak-assertion feedback, transcripts, human review), generalizing rather than overfitting, keeping it lean, explaining the why, and bundling repeated work into scripts. Rerun into

iteration-<N+1>/

and compare. Stop when results satisfy the user, feedback is empty, or gains plateau. For "is the new version actually better?", use the blind comparison described in

references/methodology.md

根据信号（失败的断言、薄弱断言的反馈、对话记录、人工评估）改进skill，注重泛化而非过拟合，保持精简，解释“为什么”，并将重复工作整合到脚本中。重新运行并保存到

iteration-<N+1>/

目录进行对比。当结果满足用户需求、无反馈或收益停滞时停止。若要判断“新版本是否真的更好”，请使用

references/methodology.md

中描述的盲态对比方法。

Outputs to report

需报告的输出结果

Layer 1: checks passed / assertions passed; any doc↔code drift found.
Layer 2: trigger pass rate (positives that fired, negatives that stayed quiet).
Layer 3: with-skill vs. without-skill pass rate delta, plus time/token cost.

第1层：通过的检查数/总检查数；通过的断言数/总断言数；是否发现文档与代码不一致的情况。
第2层：触发通过率（成功触发的正例数，保持静默的反例数）。
第3层：带Skill与无Skill的通过率差值，以及时间/令牌成本。

Reference files

参考文件

```
references/adapters.md
```
— per-CLI headless command, skills dir, auth, caveats.
```
references/methodology.md
```
— the rigorous practices (read for non-trivial evals).
```
references/grader.md
```
— how to grade a run into
```
grading.json
```
.
```
references/schemas.md
```
— exact JSON shapes for every file.
```
eval-viewer/generate_review.py
```
— render a benchmark into a standalone HTML review.

```
references/adapters.md
```
— 各CLI的无头命令、skills目录、认证信息、注意事项。
```
references/methodology.md
```
— 严谨的实践方法（非 trivial 评估请阅读）。
```
references/grader.md
```
— 如何为运行结果评分并生成
```
grading.json
```
。
```
references/schemas.md
```
— 所有文件的精确JSON格式。
```
eval-viewer/generate_review.py
```
— 将基准测试结果渲染为独立的HTML评估报告。

Verification checklist

验证清单

Do not report a verdict until each item that applies to the layers you ran is satisfied:

Layer 1: ran
```
run_script_checks.py --json
```
, recorded the
```
summary
```
line (
```
checks_passed/checks
```
,
```
assertions_passed/assertions
```
), and confirmed
```
ok: true
```
(process exit 0) — a non-zero exit means doc↔code drift, not a passing skill.
Layer 1: for at least one numeric assertion, re-derived the expected value by hand and confirmed the script's emitted value matches it (e.g.
```
approx
```
within the stated
```
rel_tol
```
/
```
abs_tol
```
) — not merely that the assertion's
```
passed
```
flag is true.
Layer 1: recorded
```
cases_without_checks
```
; if any computable eval lacks a
```
script_check
```
, noted it as a coverage gap rather than treating the run as fully verified.
Layer 2: ran
```
run_trigger_eval.py
```
with a labelled set containing both positives AND tricky negatives, and recorded the per-class pass counts (positives that fired at rate ≥ threshold, negatives that stayed below) — a positives-only run measures recall, not discrimination.
Layer 2: used
```
--runs-per-query
```
≥ 3 and recorded each query's
```
trigger_rate
```
; flagged any query whose rate sits near the
```
--threshold
```
as unstable rather than counting it as a clean pass/fail.
Layer 3: ran BOTH
```
with_skill
```
and a
```
without_skill
```
baseline, then reported
```
run_summary.delta.pass_rate
```
(the headline value) with mean ± stddev — never an absolute with-skill pass rate alone.
Layer 3: graded each run from the actual files in its
```
outputs/
```
(re-deriving numbers / opening artifacts per
```
references/grader.md
```
), recorded concrete
```
evidence
```
per expectation, and put the outputs or
```
benchmark.md
```
in front of the user before concluding.

在给出结论前，确保已完成所有适用于你所运行层级的检查项：

第1层：运行了
```
run_script_checks.py --json
```
，记录了
```
summary
```
行（
```
checks_passed/checks
```
、
```
assertions_passed/assertions
```
），并确认
```
ok: true
```
（进程退出码为0）——非零退出码表示存在文档与代码不一致的情况，skill未通过测试。
第1层：至少对一个数值断言手动推导了预期值，并确认脚本输出的数值与之匹配（例如
```
approx
```
在指定的
```
rel_tol
```
/
```
abs_tol
```
范围内）——而不仅仅是断言的
```
passed
```
标记为true。
第1层：记录了
```
cases_without_checks
```
；如果任何可计算的评估项缺少
```
script_check
```
，需将其记录为覆盖缺口，而非将该运行视为完全验证通过。
第2层：使用包含正例和有挑战性反例的带标签查询集运行了
```
run_trigger_eval.py
```
，并记录了每个类别的通过数（触发率≥阈值的正例，触发率低于阈值的反例）——仅包含正例的运行只能衡量召回率，无法衡量判别能力。
第2层：使用了
```
--runs-per-query
```
≥3，并记录了每个查询的
```
trigger_rate
```
；将任何触发率接近
```
--threshold
```
的查询标记为不稳定，而非将其视为明确的通过/失败。
第3层：同时运行了
```
with_skill
```
和
```
without_skill
```
基准测试，然后报告了
```
run_summary.delta.pass_rate
```
（核心指标）及均值±标准差——绝不能仅报告带Skill的绝对通过率。
第3层：根据每个运行的
```
outputs/
```
中的实际文件进行评分（根据
```
references/grader.md
```
重新推导数值/查看产物），为每个预期记录了具体的
```
evidence
```
，并在得出结论前将输出结果或
```
benchmark.md
```
展示给用户。

Common pitfalls & rationalizations

常见误区与合理化借口

Tempting shortcut	Why it's wrong / what to do
"The script ran and exited 0, so the skill is correct."	Exit 0 only means the process did not crash; `run_script_checks.py` returns non-zero only when an assertion or `expect_exit` fails. Read the `assertions_passed/assertions` count and re-derive at least one number — a script can run fine and still emit the wrong value.
"The assertion passed, so the number is right."	A weak assertion (e.g. "mentions cfl_checker.py", or `exists` / `truthy` on a field) passes even for a wrong run. Use value+conclusion assertions ( `approx` / `eq` with a re-derived expected), and act on the grader's `eval_feedback` that flags trivially-satisfiable assertions.
"All my trigger queries fired, so discovery works."	A positives-only set measures recall, not precision; an over-eager description that triggers on everything also passes. You need tricky near-miss negatives that stay below `--threshold` — without them the discrimination test is meaningless.
"One run per query is enough to read the trigger rate."	Detection is a heuristic over the transcript and triggering is stochastic; a single run gives a 0/1 rate. Use `--runs-per-query` ≥ 3 and treat rates hovering at the threshold as unstable, not decisive.
"With-skill pass rate is high, so the skill is valuable."	Value is the with/without delta, not the absolute rate. If the agent already aces the task without the skill, the delta is ~0 and the skill may only add latency/tokens. Always run the `without_skill` baseline and report `delta.pass_rate` .
"Assertions passed, no need to open the output files."	Automated grading only checks what you thought to assert, and a transcript can claim work it did not do. Open the files in `outputs/` , re-derive the numbers, read `user_notes.md` , and review `benchmark.md` (or have the user review it) before declaring the skill good.

诱人的捷径	为什么错误/正确做法
“脚本运行并返回0，所以skill是正确的。”	返回0仅表示进程未崩溃；只有当断言或 `expect_exit` 失败时， `run_script_checks.py` 才会返回非零退出码。请查看 `assertions_passed/assertions` 的计数，并至少手动推导一个数值——脚本可能运行正常但输出错误的数值。
“断言通过了，所以数值是正确的。”	薄弱的断言（例如“提到了cfl_checker.py”，或对某个字段使用 `exists` / `truthy` ）即使在运行错误的情况下也会通过。请使用包含值和结论的断言（ `approx` / `eq` 搭配手动推导的预期值），并根据评分器的 `eval_feedback` 处理那些容易满足的断言。
“我所有的触发查询都触发了，所以发现功能正常。”	仅包含正例的集合只能衡量召回率，无法衡量精确率；一个过于宽泛的描述会在所有情况下触发，也能通过该测试。你需要添加有挑战性的近似反例，使其触发率低于 `--threshold` ——没有这些反例，判别测试毫无意义。
“每个查询运行一次就足以得出触发率。”	触发检测是基于对话记录的启发式方法，且触发具有随机性；单次运行只能得到0/1的结果。请使用 `--runs-per-query` ≥3，并将触发率接近阈值的查询视为不稳定，而非决定性结果。
“带Skill的通过率很高，所以skill很有价值。”	价值在于有无Skill的差值，而非绝对通过率。如果Agent在无Skill的情况下就能完美完成任务，那么差值约为0，该Skill可能只会增加延迟/令牌消耗。务必运行 `without_skill` 基准测试并报告 `delta.pass_rate` 。
“断言都通过了，无需查看输出文件。”	自动评分仅检查你设定的断言，而对话记录可能会声称完成了未实际执行的工作。请打开 `outputs/` 中的文件，重新推导数值，阅读 `user_notes.md` ，并在宣布skill合格前查看 `benchmark.md` （或让用户查看）。

Security

安全性

Input Validation

输入验证

```
--agent
```
is resolved against a fixed allowlist of known adapter ids/aliases (
```
agent_adapters.py
```
); unknown values are rejected (exit 2).
```
--skill
```
must be a directory containing
```
SKILL.md
```
or the runners exit 2.
```
script_checks
```
operators and dotted paths are matched against fixed sets; no user string is ever
```
eval()
```
'd or passed to a shell.

```
--agent
```
会与已知适配器ID/别名的固定白名单（
```
agent_adapters.py
```
）进行匹配；未知值会被拒绝（退出码2）。
```
--skill
```
必须是包含
```
SKILL.md
```
的目录，否则运行器会返回退出码2。
```
script_checks
```
的操作符和点路径会与固定集合进行匹配；绝不会对用户输入的字符串执行
```
eval()
```
或传递给shell。

File Access

文件访问

The deterministic layer runs a skill's own scripts with the real interpreter and reads only that skill's
```
evals/evals.json
```
.
The quality/trigger layers create isolated working directories under a user-supplied workspace, copy the skill into them, and write results there.

确定性层使用真实解释器运行skill自身的脚本，仅读取该skill的
```
evals/evals.json
```
。
质量/触发层会在用户提供的工作区下创建独立的工作目录，将skill复制到其中，并将结果写入该目录。

Tool Restrictions

工具限制

Bash: runs the harness Python scripts and the selected coding-agent CLI.
Read/Grep/Glob: inspect skills and results. Write: scaffold workspaces.

Bash：运行测试框架的Python脚本和所选的编码Agent CLI。
Read/Grep/Glob：检查skill和结果。Write：搭建工作区。

Safety Measures

安全措施

No
```
eval()
```
/
```
exec()
```
; subprocess calls use explicit argument lists (never
```
shell=True
```
); commands are built from the adapter spec, not string-concatenated.
The trigger/quality layers pass each CLI's auto-approve flag (e.g.
```
--dangerously-skip-permissions
```
), which runs the agent with reduced safeguards. Only evaluate skills you trust, ideally inside a sandbox/container. Always
```
--dry-run
```
first to inspect the exact command. Auth is read from environment variables, never passed as command arguments.

不使用
```
eval()
```
/
```
exec()
```
；子进程调用使用显式参数列表（绝不使用
```
shell=True
```
）；命令根据适配器规范构建，而非字符串拼接。
触发/质量层会传递每个CLI的自动批准标志（例如
```
--dangerously-skip-permissions
```
），这会以降低安全防护的方式运行Agent。仅评估你信任的skill，理想情况下在沙箱/容器内运行。务必先执行
```
--dry-run
```
以查看具体命令。认证信息从环境变量读取，绝不会作为命令参数传递。

Limitations

局限性

Layers 2–3 require a supported CLI installed and authenticated; otherwise use Layer 1 only.
Trigger detection is a cross-tool heuristic (did the transcript consult the skill?); for the most precise detection on Claude Code, parse its stream-json tool-use events.
Token accounting is best-effort — only some CLIs report usage in headless output.
New CLIs (Antigravity, Grok) are medium confidence; verify flags with the vendor
```
--help
```
and
```
--dry-run
```
.

第2-3层需要安装并认证支持的CLI；否则只能使用第1层。
触发检测是跨工具的启发式方法（对话记录是否参考了skill？）；要在Claude Code上实现最精确的检测，请解析其流式JSON工具使用事件。
令牌统计是尽力而为的——只有部分CLI会在无头输出中报告使用情况。
新CLI（Antigravity、Grok）的可信度为中等；请使用供应商的
```
--help
```
和
```
--dry-run
```
验证标志。