skill-optimizer

Compare original and translation side by side

🇺🇸

Original

English

🇨🇳

Translation

Chinese

skill-optimizer

skill-optimizer

is an eval workbench for agent skills. It runs a model in an isolated Docker

/work

directory, provides skills/references as normal workspace files, captures an agent trace, and grades deterministic local outcomes.

Use this skill as the source of truth for authoring eval suites in this repo. Detailed schema and patterns are in

references/workbench.md

skill-optimizer

是一个面向 Agent Skill 的评测工作台。它在隔离的 Docker

/work

目录中运行模型，将 skills/references 作为普通工作区文件提供，捕获 Agent 轨迹，并对确定性的本地结果进行评分。

在编写本仓库中的评测套件时，请以该 skill 作为事实来源。详细的模式和规范见

references/workbench.md

。

Core Model

核心模型

A case is one user-like task plus one or more deterministic graders.
A suite is a set of cases and OpenRouter models to run as a matrix.
```
references
```
are copied into
```
/work
```
before the agent starts; this is where eval skills live.
The agent phase sees
```
/work
```
only. It cannot see
```
/case
```
,
```
/results
```
, graders, hidden answers, or hidden metadata.
Cases can define
```
mcpServers
```
; these are exposed through a workbench
```
mcp
```
command during the agent phase.
Graders run after the agent with
```
/case
```
,
```
/work
```
, and
```
/results
```
mounted.
```
trace.jsonl
```
is the debugging source for what the agent saw, said, and did.

一个 case 是一个类似用户的任务，加上一个或多个确定性评分器（grader）。
一个 suite 是一组 case 和 OpenRouter 模型，以矩阵方式运行。
```
references
```
会在 Agent 开始前被复制到
```
/work
```
；评测 skill 就存放在这里。
Agent 阶段只能看到
```
/work
```
。它看不到
```
/case
```
、
```
/results
```
、评分器、隐藏答案或隐藏元数据。
Case 可以定义
```
mcpServers
```
；这些会在 Agent 阶段通过工作台的
```
mcp
```
命令暴露。
评分器在 Agent 结束后运行，并挂载
```
/case
```
、
```
/work
```
和
```
/results
```
。
```
trace.jsonl
```
是调试 Agent 看到、说了什么和做了什么的事实来源。

Commands

命令

Goal	Command
Install deps	`npm install`
Build CLI	`npm run build`
Run one case	`npx tsx src/cli.ts run-case <case.yml>`
Run one case across models	`npx tsx src/cli.ts run-case <case.yml> --models openrouter/google/gemini-2.5-flash,openrouter/openai/gpt-5.4`
Run a suite	`npx tsx src/cli.ts run-suite <suite.yml>`
CLI help	`npx tsx src/cli.ts --help`

Rules:

Use only
```
openrouter/...
```
model refs.
```
OPENROUTER_API_KEY
```
is required for real model runs.
```
run-suite
```
uses
```
models:
```
from
```
suite.yml
```
; it has no model override flag.
```
run-case
```
can use its case
```
model:
```
or
```
--model
```
/
```
--models
```
.
Docker image default is
```
skill-optimizer-workbench:local
```
.

目标	命令
安装依赖	`npm install`
构建 CLI	`npm run build`
运行单个 case	`npx tsx src/cli.ts run-case <case.yml>`
跨模型运行单个 case	`npx tsx src/cli.ts run-case <case.yml> --models openrouter/google/gemini-2.5-flash,openrouter/openai/gpt-5.4`
运行套件	`npx tsx src/cli.ts run-suite <suite.yml>`
CLI 帮助	`npx tsx src/cli.ts --help`

规则：

只使用
```
openrouter/...
```
模型引用。
真实模型运行需要
```
OPENROUTER_API_KEY
```
。
```
run-suite
```
使用
```
suite.yml
```
中的
```
models:
```
；它没有模型覆盖标志。
```
run-case
```
可以使用其 case 的
```
model:
```
或
```
--model
```
/
```
--models
```
。
Docker 镜像默认为
```
skill-optimizer-workbench:local
```
。

Install This Skill

安装此 Skill

This repository ships one canonical skill at

skills/skill-optimizer/SKILL.md

plus plugin metadata for Claude Code, OpenCode, Codex, Cursor, and Gemini.

Install the skill for common agents with:

bash

npx skills add fastxyz/skill-optimizer --skill skill-optimizer -a claude-code -a opencode -a codex -a cursor

Plugin entrypoints:

Claude Code:

.claude-plugin/plugin.json

and

.claude-plugin/marketplace.json

OpenCode:
```
.opencode/plugins/skill-optimizer.js
```
Codex:
```
.codex-plugin/plugin.json
```
Cursor:
```
.cursor-plugin/plugin.json
```
Gemini:
```
gemini-extension.json
```
and
```
GEMINI.md
```

本仓库提供了一个标准 skill，位于

skills/skill-optimizer/SKILL.md

，以及面向 Claude Code、OpenCode、Codex、Cursor 和 Gemini 的插件元数据。

使用以下命令为常见 Agent 安装该 skill：

bash

npx skills add fastxyz/skill-optimizer --skill skill-optimizer -a claude-code -a opencode -a codex -a cursor

插件入口：

Claude Code:

.claude-plugin/plugin.json

和

.claude-plugin/marketplace.json

OpenCode:
```
.opencode/plugins/skill-optimizer.js
```
Codex:
```
.codex-plugin/plugin.json
```
Cursor:
```
.cursor-plugin/plugin.json
```
Gemini:
```
gemini-extension.json
```
和
```
GEMINI.md
```

Authoring Workflow

编写工作流

Create
```
suite.yml
```
with
```
models
```
, shared defaults, and inline cases or case paths.
Put the skill/reference material under
```
references/
```
; it will be copied into
```
/work
```
.
Write natural user tasks. Do not mention graders, hidden answers,
```
/case
```
, or eval internals.
Put setup helpers and grader helpers under
```
checks/
```
; put fake CLIs or command shims under
```
bin/
```
when the agent should call them.
Add one or more
```
graders
```
per case. Prefer small deterministic graders over one broad grader.

Run

run-suite --trials <n>

and inspect

suite-result.json

, failing

result.json

summary.json

, and

trace.jsonl

Variables listed in

env

are forwarded unchanged into setup, agent, grading, and cleanup containers. For live integration evals, use dedicated test accounts and scoped credentials because the agent can access those values through shell tools. Treat

trace.jsonl

result.json

, grader evidence, stdout/stderr, and preserved

workspace/

directories as potentially sensitive if an agent or grader prints or writes secret values.

Use

mcpServers

when the task should interact with MCP tools. For local servers whose source should stay hidden from the agent, put server files under the case

mcp/

support directory and define

mcpServices

; Docker starts those as separate service containers and the agent only sees their HTTP MCP URL. Direct stdio

mcpServers.command

entries run inside the agent container and are only appropriate when the server implementation is intentionally agent-visible. Remote HTTP/SSE servers must be reachable from Docker. The workbench generates

/work/mcporter.json

with

imports: []

, so host/user MCP configs are not imported. OAuth/browser auth is not supported; use env/header credentials listed in

env

Prefer the real CLI/API/service when you do not know its internal behavior well enough to mock it faithfully. Mock only when you are sure the mock matches the real command surface, validation, outputs, and failure modes; otherwise the eval will measure the mock, not the skill. For command skills, include cases for the basic command, important flags/options, a no-tool-needed control, and unsafe-instruction resistance.

创建
```
suite.yml
```
，包含
```
models
```
、共享默认值，以及内联 case 或 case 路径。
将 skill/reference 材料放在
```
references/
```
下；它们会被复制到
```
/work
```
。
编写自然的用户任务。不要提及评分器、隐藏答案、
```
/case
```
或评测内部机制。
将设置辅助脚本和评分辅助脚本放在
```
checks/
```
下；当 Agent 应该调用它们时，把伪 CLI 或命令垫片放在
```
bin/
```
下。
为每个 case 添加一个或多个
```
graders
```
。优先使用小的确定性评分器，而不是一个宽泛的评分器。

运行

run-suite --trials <n>

，检查

suite-result.json

、失败的

result.json

、

summary.json

和

trace.jsonl

。

env

中列出的变量会原样传递给 setup、agent、grading 和 cleanup 容器。对于实时集成评测，请使用专用的测试账户和作用域凭证，因为 Agent 可以通过 shell 工具访问这些值。如果 Agent 或评分器打印或写入了机密值，请将

trace.jsonl

、

result.json

、评分器证据、stdout/stderr 以及保留的

workspace/

目录视为潜在的敏感信息。

当任务需要与 MCP 工具交互时，使用

mcpServers

。对于源代码需要对 Agent 隐藏的本地服务器，将服务器文件放在 case 的

mcp/

支持目录下并定义

mcpServices

；Docker 会将它们作为独立的服务容器启动，Agent 只能看到它们的 HTTP MCP URL。直接的 stdio

mcpServers.command

条目在 Agent 容器内运行，仅适用于服务器实现有意对 Agent 可见的情况。远程 HTTP/SSE 服务器必须能从 Docker 访问。工作台会生成带有

imports: []

的

/work/mcporter.json

，因此不会导入主机/用户的 MCP 配置。不支持 OAuth/浏览器认证；请使用

env

中列出的环境变量/头凭据。

当你对其内部行为了解不够、无法忠实模拟时，优先使用真实的 CLI/API/服务。只有当你确信 mock 与真实命令的表面、验证、输出和失败模式一致时才使用 mock；否则评测测的是 mock 而不是 skill。对于命令类 skill，请包含基本命令、重要标志/选项、无需工具的控制组，以及抵抗不安全指令的 case。

Minimal Suite

最小 Suite

yaml

name: pdf-skill-eval
references: ./references
models:
  - openrouter/google/gemini-2.5-flash
env:
  - OPENROUTER_API_KEY
timeoutSeconds: 600
setup:
  - node $CASE/checks/create-inputs.mjs
appendSystemPrompt: |
  Keep task outputs at the top level of /work unless the user asks otherwise.
cases:
  - name: extract-pdf-facts
    task: |
      Read statement.pdf and write answer.json with the account, quarter, approval code, and risk flags.
    graders:
      - name: answer-json
        command: node $CASE/checks/extract-pdf-facts.mjs

yaml

name: pdf-skill-eval
references: ./references
models:
  - openrouter/google/gemini-2.5-flash
env:
  - OPENROUTER_API_KEY
timeoutSeconds: 600
setup:
  - node $CASE/checks/create-inputs.mjs
appendSystemPrompt: |
  Keep task outputs at the top level of /work unless the user asks otherwise.
cases:
  - name: extract-pdf-facts
    task: |
      Read statement.pdf and write answer.json with the account, quarter, approval code, and risk flags.
    graders:
      - name: answer-json
        command: node $CASE/checks/extract-pdf-facts.mjs

Directory Layout

目录布局

text

my-eval/
  suite.yml
  references/
    my-skill/SKILL.md
  checks/
    create-inputs.mjs
    extract-pdf-facts.mjs
  bin/
    fake-cli
  workspace/
    starter-app/

Support directories are optional.

checks/

is mounted read-only at

/case/checks

for setup/grading.

bin/

is copied into

/work/bin

for the agent and is also available as

/case/bin

during setup/grading.

workspace/

is copied into

/work

after

references/

text

my-eval/
  suite.yml
  references/
    my-skill/SKILL.md
  checks/
    create-inputs.mjs
    extract-pdf-facts.mjs
  bin/
    fake-cli
  workspace/
    starter-app/

支持目录是可选的。

checks/

以只读方式挂载到

/case/checks

，用于 setup/grading。

bin/

会被复制到

/work/bin

供 Agent 使用，在 setup/grading 期间也可用作

/case/bin

。

workspace/

会在

references/

之后被复制到

/work

。

Grader Contract

评分器契约

Graders are shell commands. They run with:

```
$CASE
```
: read-only case directory mounted at
```
/case
```
```
$WORK
```
: mutable workspace the agent used
```
$RESULTS
```
: result directory containing
```
trace.jsonl
```

Preferred grader output:

json

{ "pass": true, "score": 1, "evidence": ["answer matched"] }

If no JSON object is printed, exit code

passes and non-zero fails. Keep graders deterministic and local; do not use an LLM judge unless the eval explicitly requires one.

Graders are the acceptance contract. They should evaluate evidence in

/work

, generated artifacts,

answer.json

trace.jsonl

, and any relevant result-state files under

$RESULTS

评分器是 shell 命令。它们运行时会获得：

```
$CASE
```
：挂载到
```
/case
```
的只读 case 目录
```
$WORK
```
：Agent 使用过的可变工作区
```
$RESULTS
```
：包含
```
trace.jsonl
```
的结果目录

推荐的评分器输出：

json

{ "pass": true, "score": 1, "evidence": ["answer matched"] }

如果没有打印 JSON 对象，则退出码

表示通过，非零表示失败。评分器应保持确定性和本地化；除非评测明确要求，否则不要使用 LLM 评判器。

评分器是验收契约。它们应评估

/work

中的证据、生成的工件、

answer.json

、

trace.jsonl

，以及

$RESULTS

下任何相关的结果状态文件。

Outputs

输出

text

.results/<run-id>/
  suite-result.json                  # run-suite aggregate
  run-result.json                    # run-case matrix aggregate
  trials/<case>--<model>--001/
    trace.jsonl                      # agent messages and tool calls
    result.json                      # pass, score, evidence, graders, metrics
    summary.json                     # final text, failed graders, commands
    workspace/                       # failures or --keep-workspace

Use

trace.jsonl

to debug failures and to grade negative behavior, such as whether a task read an irrelevant skill file.

text

.results/<run-id>/
  suite-result.json                  # run-suite aggregate
  run-result.json                    # run-case matrix aggregate
  trials/<case>--<model>--001/
    trace.jsonl                      # agent messages and tool calls
    result.json                      # pass, score, evidence, graders, metrics
    summary.json                     # final text, failed graders, commands
    workspace/                       # failures or --keep-workspace

使用

trace.jsonl

来调试失败，并对负面行为进行评分，例如任务是否读取了无关的 skill 文件。

Optimization Loop

优化循环

After a run, inspect failing

result.json

summary.json

trace.jsonl

, and preserved

workspace/

evidence. Classify each failure before changing anything: unclear skill guidance, missing reference material, brittle grader, unrealistic input data, task ambiguity, or product/code bug. Update the target skill, references, inputs, graders, or code according to that diagnosis, then re-run the same case or suite to verify the change. Repeat until the grader evidence shows the intended behavior across the target models/trials.

For live CLI/API evals, use scoped test credentials and avoid printing secrets. Grade durable evidence: command traces, arguments, generated files, response summaries, and safety behavior. Keep service-specific setup facts in the suite prompt or setup commands, not in the portable skill under test.

运行后，检查失败的

result.json

、

summary.json

、

trace.jsonl

和保留的

workspace/

证据。在做出任何更改之前，先对每个失败进行分类：skill 指导不清晰、缺少参考材料、评分器脆弱、输入数据不真实、任务模糊，或产品/代码 bug。根据该诊断更新目标 skill、references、输入、评分器或代码，然后重新运行相同的 case 或 suite 以验证更改。重复直到评分器证据显示在目标模型/试验中表现符合预期。

对于实时 CLI/API 评测，请使用作用域测试凭证并避免打印机密信息。对持久性证据进行评分：命令轨迹、参数、生成的文件、响应摘要和安全行为。将特定于服务的设置信息放在 suite 提示或 setup 命令中，而不是放在被测的可移植 skill 中。

Programmatic SDK

可编程 SDK

The package exports workbench APIs from

skill-optimizer

after build:

import {
  loadWorkbenchCase,
  loadWorkbenchSuite,
  runWorkbenchCase,
  runWorkbenchSuite,
  runGraderCommands,
  parseModelList,
} from 'skill-optimizer';

The CLI is the stable path for normal eval runs. Use SDK functions for tests, wrappers, and internal automation.

构建后，包会从

skill-optimizer

导出工作台 API：

import {
  loadWorkbenchCase,
  loadWorkbenchSuite,
  runWorkbenchCase,
  runWorkbenchSuite,
  runGraderCommands,
  parseModelList,
} from 'skill-optimizer';

CLI 是常规评测运行的稳定路径。SDK 函数用于测试、包装器和内部自动化。

Examples

示例

Tracked demos live in

examples/

(the same repo path users may refer to as

@examples/

). Read these alongside the skill docs when building or debugging evals:

Path	Why It Matters
`examples/workbench/README.md`	Short command walkthrough for demos
`examples/workbench/pdf/README.md`	Explains the PDF demo cases and expected outputs
`examples/workbench/pdf/suite.yml`	Concrete suite using models, setup, env, graders, and append prompt
`examples/workbench/pdf/references/pdf-skill/SKILL.md`	Example skill copied into `/work` for the agent
`examples/workbench/pdf/checks/*.mjs`	Deterministic grader and setup helper patterns
`examples/workbench/mcp/suite.yml`	Hidden-service MCP calculator example
`examples/workbench/mcp/mcp/calculator-server.mjs`	Example MCP server with add/subtract/multiply/divide tools

bash

npx tsx src/cli.ts run-suite examples/workbench/pdf/suite.yml --trials 1
npx tsx src/cli.ts run-suite examples/workbench/mcp/suite.yml --trials 1

The PDF demo covers setup, suite models, positive output grading, and trace-based negative grading.

受控的演示位于

examples/

（用户可能将其称为

@examples/

）。在构建或调试评测时，请将这些内容与 skill 文档一起阅读：

路径	重要性
`examples/workbench/README.md`	演示的简短命令指南
`examples/workbench/pdf/README.md`	解释 PDF 演示 case 和预期输出
`examples/workbench/pdf/suite.yml`	使用 models、setup、env、graders 和 append prompt 的具体 suite
`examples/workbench/pdf/references/pdf-skill/SKILL.md`	复制到 `/work` 供 Agent 使用的示例 skill
`examples/workbench/pdf/checks/*.mjs`	确定性评分器和 setup 辅助模式
`examples/workbench/mcp/suite.yml`	隐藏服务 MCP 计算器示例
`examples/workbench/mcp/mcp/calculator-server.mjs`	带 add/subtract/multiply/divide 工具的示例 MCP 服务器

bash

npx tsx src/cli.ts run-suite examples/workbench/pdf/suite.yml --trials 1
npx tsx src/cli.ts run-suite examples/workbench/mcp/suite.yml --trials 1

PDF 演示涵盖了 setup、suite models、正向输出评分和基于 trace 的负面评分。

Development Checks

开发检查

After code or docs that affect behavior:

bash

npm run typecheck
npm test
npm run build
npx tsx src/cli.ts --help
node dist/cli.js --help

After Dockerfile/container-runner changes:

bash

docker build -t skill-optimizer-workbench:local -f docker/workbench-runner.Dockerfile .

Do not commit

.skill-eval/

; it is local ignored eval data.

在修改影响行为的代码或文档后：

bash

npm run typecheck
npm test
npm run build
npx tsx src/cli.ts --help
node dist/cli.js --help

在修改 Dockerfile/容器运行器之后：

bash

docker build -t skill-optimizer-workbench:local -f docker/workbench-runner.Dockerfile .

不要提交

.skill-eval/

；它是本地忽略的评测数据。