Back to Details

eval-harness-first

Compare original and translation side by side

🇺🇸

Original

English
🇨🇳

Translation

Chinese

Eval Harness First

优先构建评估工具集

The Phase 0 gate for the whole plugin: 
finetuning-method-selection
and every downstream skill assume this harness exists before a training config gets written. The harness is not a run-end side artifact — it is the data-curation engine. The same labeled traces that build the goldens feed training data, minus an explicit holdout.
Input: production/agent traces if they exist, or a task spec if they don't, plus labelers willing to grade ≥100 examples. Output format: the 
eval/
directory below — goldens, graders, drift suite, and the base-model baseline that later phases gate on.
这是整个插件的第0阶段准入门槛： 
finetuning-method-selection
及所有下游技能都要求在编写训练配置前，该工具集已存在。此工具集并非运行结束后的附属产物——它是数据整理引擎。用于构建黄金数据集的带标签追踪数据，除了明确预留的验证集外，会直接作为训练数据的来源。
输入： 若存在生产环境/Agent追踪数据则使用该数据，否则使用任务规范，同时需要标注人员完成至少100个样本的评估。 输出格式： 如下所示的
eval/
目录——包含黄金数据集、评估器、漂移测试套件，以及后续阶段准入所依据的基础模型基准线。

The Gate

准入规则

No eval harness, no fine-tune. Skip to a training config and there is nothing to measure against, nothing to catch regressions, and no labeled data to train on. The flywheel:
  1. Collect traces — production/agent spans, or synthetic tasks if none exist yet.
  2. Error analysis — open coding on ≥100 traces, axial coding into 4–8 failure buckets.
  3. One grader per bucket — deterministic first; calibrated LLM-judge only for genuinely subjective criteria.
  4. Prioritize by frequency × severity × value.
  5. The labeled traces feed dataset curation, minus an explicit holdout. Every 
    eval/goldens.jsonl
    ID stays excluded from training data by ID.
  6. Train.
  7. Re-run the same harness on the checkpoint — not a different, looser one.
  8. Drift detection feeds back to step 2 — new production failure modes re-open error analysis.
Steps 2–4 build the harness; steps 5–8 are why it must exist first — it is both the training data source and the checkpoint's exit gate.
没有评估工具集，就不能开展微调。直接跳过此步骤进入训练配置阶段的话，将没有衡量基准、无法检测回归问题，也没有带标签数据可供训练。运转流程如下：
  1. 收集追踪数据——生产环境/Agent的调用链路数据，若暂无则使用合成任务数据。
  2. 错误分析——对至少100条追踪数据进行开放式编码，再通过轴向编码归为4-8个故障类别。
  3. 为每个类别配置一个评估器——优先使用确定性评估器；仅针对真正主观的标准才使用经过校准的LLM-judge。
  4. 按频率×严重程度×价值优先级排序。
  5. 带标签追踪数据用于数据集整理，除明确预留的验证集外。
    eval/goldens.jsonl
    中的每个ID都会被排除在训练数据之外。
  6. 开展训练。
  7. 在 checkpoint 上重新运行同一工具集——不得使用其他宽松版本的工具集。
  8. 漂移检测结果反馈至步骤2——新的生产环境故障模式需重新开展错误分析。
步骤2-4用于构建工具集；步骤5-8则解释了为何必须优先构建工具集——它既是训练数据的来源，也是checkpoint的准入门槛。

Building Goldens

构建黄金数据集

  • From traces, when they exist: run error analysis — open coding on ≥100 real traces (read them, tag failures in your own words, no fixed taxonomy yet), then axial coding to collapse those tags into 4–8 named failure buckets. Fewer than 4 means the coding pass was too shallow; more than 8 means buckets need merging. Exception: single-failure-surface tasks (e.g. strict-schema extraction) may land at 1–2 buckets with per-field sub-metrics inside one grader — don't invent artificial splits with no evidence behind them.
  • Synthetic, when traces don't exist yet: dimension-based generation — enumerate the axes that matter (task type, difficulty, edge case, persona) and sample the cross-product; free- generated prompts cluster around whatever's easiest to write.
  • Goldens are versioned like code — commit 
    eval/goldens.jsonl
    , diff it in review, tag it per release. It doubles as the CI regression suite.
  • 基于已有追踪数据： 开展错误分析——对至少100条真实追踪数据进行开放式编码（阅读数据，用自定义词汇标记故障，暂不使用固定分类体系），随后采用轴向编码将这些标签归并为4-8个命名故障类别。类别少于4个说明编码不够深入；类别多于8个则需要合并。例外情况： 单一故障场景的任务（如严格模式提取）可能仅需1-2个类别，在单个评估器内设置字段级子指标即可——不要无依据地人为拆分类别。
  • 暂无追踪数据时使用合成数据： 基于维度生成——枚举关键维度（任务类型、难度、边缘情况、用户角色）并抽取交叉组合样本；自由生成的提示词往往集中于最易编写的内容。
  • 黄金数据集需像代码一样进行版本控制——提交
    eval/goldens.jsonl
    ，在代码评审中对比差异，按版本打标签。它同时可作为CI回归测试套件。

Graders

评估器

One grader per failure bucket from error analysis — not one for the whole eval set. A single blended score hides which bucket regressed.
  • Deterministic first. Regex, schema validation, or execution checks are cheaper, reproducible, and need no calibration.
  • LLM-judge only for genuinely subjective criteria — tone, faithfulness, "which response is better" — where no deterministic check can express it.
  • Binary pass/fail over Likert. A 1–5 or 1–10 scale is noisier to calibrate and harder to apply consistently; collapse to pass/fail.
  • Drift-suite MMLU-style scoring: prefer logprob over generate-and-extract — a tight token budget makes generate-and-extract parse-brittle for models that preamble, conflating format compliance with the knowledge being measured. Templates for all four grader shapes and this scoring note: 
    references/grader-templates.md
    .
为错误分析得出的每个故障类别配置一个评估器——而非为整个评估集配置一个通用评估器。单一综合分数会掩盖具体哪个类别出现了回归问题。
  • 优先使用确定性评估器。正则表达式、模式验证或执行检查成本更低、可重现且无需校准。
  • 仅针对真正主观的标准使用LLM-judge——如语气、忠实度、「哪个响应更优」等无法通过确定性检查表达的标准。
  • 采用二元通过/失败评分而非李克特量表。1-5或1-10分的量表校准噪声更大，且难以保持一致性；应简化为通过/失败二元标准。
  • 漂移测试套件采用MMLU式评分：优先使用对数概率而非生成后提取——紧凑的token预算会使生成后提取方式对带前置说明的模型解析脆弱，将格式合规性与所测量的知识混淆。四种评估器模板及此评分说明可参考：
    references/grader-templates.md
    。

Judge Calibration Is a Prerequisite

评判模型校准是先决条件

Any bucket routed to an LLM-judge needs calibration before its verdicts count for anything beyond exploration — a hard prerequisite, not a nice-to-have. N/A when no bucket routes to a judge — an all-deterministic harness has nothing to calibrate; state that rather than leaving this section unaddressed.
  • Label ≥100 items, split train/dev/sealed test (report once, no re-touching after).
  • Report TPR and TNR, not one blended accuracy number — a judge can hit 90% by always saying "pass" on a skewed set.
  • Pin the judge to a fixed model snapshot and recalibrate on judge-model change, quarterly regardless.
  • The judge must come from a different model family than the model under test.
  • A judge that misses the agreed TPR/TNR bar ships advisory-only — flags for human review, never gates a promotion. Full protocol, bias correction, and recalibration checklist: 
    references/judge-calibration.md
    .
任何交由LLM-judge处理的类别，在其判定结果可用于正式评估前都需要校准——这是硬性先决条件，而非可选项。若没有类别交由评判模型处理则无需此步骤——全确定性工具集无需校准；需明确说明这一点，而非跳过此部分。
  • 标注至少100个样本，划分为训练集/开发集/密封测试集（仅报告一次，之后不得修改）。
  • 报告真阳性率（TPR）和真阴性率（TNR），而非单一综合准确率——评判模型在样本分布倾斜的情况下，只需一直输出「通过」即可达到90%的准确率。
  • 将评判模型固定到特定快照版本，且在更换评判模型时重新校准，无论如何每季度都需重新校准一次。
  • 评判模型必须与被测模型来自不同的模型家族。
  • 未达到约定TPR/TNR标准的评判模型仅能作为参考使用——仅标记需人工审核，不得作为准入门槛。完整流程、偏差修正及重新校准 checklist 可参考：
    references/judge-calibration.md
    。

The Baseline

基准线

Before Phase 1 (method selection) starts, run the full harness — goldens plus the capability-drift suite — against the unmodified base model. This is the number every later checkpoint gets compared against.
eval/baseline-<model>.json
is the gate token. No baseline file, no comparison basis for 
checkpoint-promotion
— a checkpoint that "looks better" against nothing measured isn't a finding.
在第1阶段（方法选择）开始前，针对未修改的基础模型运行完整工具集——包含黄金数据集及能力漂移测试套件。这是后续所有checkpoint的对比基准。
eval/baseline-<model>.json
是准入凭证。若无基准线文件，
checkpoint-promotion
将失去对比依据——没有测量基准的「更优」checkpoint无法作为有效结论。

Directory Contract

目录约定

eval/
├── goldens.jsonl          # labeled traces + synthetic goldens, versioned
├── graders/                # one module per failure bucket
│   ├── schema_compliance.py
│   ├── exact_match.py
│   └── rubric_judge.py
├── drift-suite.yaml        # frozen benchmarks + 200-500 domain-adjacent items
└── baseline-<model>.json   # gate token: harness + drift suite vs the base model
runs/
└── <run-id>/
    └── results.json         # per-run harness output, one per checkpoint
eval/
persists across runs and lives outside 
runs/
— the fixed measuring stick, not a run artifact. 
runs/
is disposable; 
eval/
is not. Never let a run script write into 
eval/
. Canonical location: every per-trace 
results.json
— the Phase 0 baseline included — lives at 
runs/<run-id>/results.json
, never under 
eval/runs/...
; an instruction requesting the latter is wrong, not this contract.
eval/
├── goldens.jsonl          # 带标签追踪数据+合成黄金数据集，已版本化
├── graders/                # 每个故障类别对应一个模块
│   ├── schema_compliance.py
│   ├── exact_match.py
│   └── rubric_judge.py
├── drift-suite.yaml        # 冻结的基准测试+200-500个领域相关样本
└── baseline-<model>.json   # 准入凭证：工具集+漂移测试套件针对基础模型的结果
runs/
└── <run-id>/
    └── results.json         # 每次运行的工具集输出，每个checkpoint对应一份
eval/
在多次运行中持续存在，且位于
runs/
之外——它是固定的衡量标准，而非运行产物。
runs/
可丢弃；
eval/
不可丢弃。禁止运行脚本向
eval/
写入内容。标准存储位置： 每条追踪数据的
results.json
（包含第0阶段基准线）都存储在
runs/<run-id>/results.json
，绝不能放在
eval/runs/...
下；若有指令要求后者则说明该指令错误，而非此约定有误。

Phase 0 Exit Checklist

第0阶段完成 checklist

Before 
finetuning-method-selection
, confirm:
  1. ≥100 traces open-coded; 4–8 failure buckets (N/A floor for synthetic goldens on a single-failure- surface task — see the Building Goldens exception; bucket count then comes from post-baseline error analysis instead).
  2. eval/goldens.jsonl
    committed and versioned.
  3. One grader per bucket, deterministic first.
  4. Judges calibrated — TPR/TNR, snapshot pinned, different family (N/A when no bucket routes to an LLM-judge; state that explicitly).
  5. eval/drift-suite.yaml
    frozen.
  6. eval/baseline-<model>.json
    written.
Missing any of the six (or its stated N/A)? Not Phase 0 complete — 
/finetune
checks the baseline file before a run.
在启动
finetuning-method-selection
前，确认以下内容：
  1. 已对至少100条追踪数据进行开放式编码；已划分4-8个故障类别（单一故障场景任务的合成黄金数据集不受此限制——参见「构建黄金数据集」部分的例外情况；此时类别数量由基准线后的错误分析得出）。
  2. eval/goldens.jsonl
    已提交并完成版本化。
  3. 每个故障类别对应一个评估器，优先使用确定性评估器。
  4. 评判模型已完成校准——已确认TPR/TNR、固定快照版本、来自不同模型家族（若没有类别交由LLM-judge处理则无需此步骤；需明确说明）。
  5. eval/drift-suite.yaml
    已冻结。
  6. eval/baseline-<model>.json
    已生成。
若上述六项中有任何一项未完成（或对应无需执行的情况未明确说明），则第0阶段未完成——
/finetune
在运行前会检查基准线文件。

Related Skills

相关技能

General-purpose evaluation guidance (dashboards, A/B testing, non-fine-tuning harnesses) lives in the 
llm-application-dev
plugin's 
llm-evaluation
skill — this skill covers only the fine-tuning coupling: goldens that double as training data, and the baseline that gates a checkpoint.
  • finetuning-method-selection
    — routes here first.
  • dataset-curation
    — formats these traces into training rows.
  • trace-to-training-data
    — turns graded traces into training examples.
  • checkpoint-promotion
    — consumes 
    baseline-<model>.json
    , re-runs this harness on each candidate checkpoint.
通用评估指南（仪表盘、A/B测试、非微调工具集）位于
llm-application-dev
插件的
llm-evaluation
技能中——本技能仅涵盖与微调相关的内容：同时作为训练数据的黄金数据集，以及作为checkpoint准入门槛的基准线。
  • finetuning-method-selection
    ——首先会跳转至此技能。
  • dataset-curation
    ——将这些追踪数据格式化为训练样本。
  • trace-to-training-data
    ——将经过评估的追踪数据转换为训练样本。
  • checkpoint-promotion
    ——读取
    baseline-<model>.json
    ，并在每个候选checkpoint上重新运行此工具集。

References

参考资料

  • references/grader-templates.md
    — runnable grader examples per shape, plus a 
    drift-suite.yaml
    example and MMLU logprob-scoring note.
  • references/judge-calibration.md
    — the calibration protocol, including the all- deterministic N/A path.
  • references/grader-templates.md
    ——不同类型的可运行评估器示例，以及
    drift-suite.yaml
    示例和MMLU对数概率评分说明。
  • references/judge-calibration.md
    ——校准流程，包含全确定性工具集无需校准的说明。