Back to Details

agent-eval-harness-ops

Compare original and translation side by side

🇺🇸

Original

English
🇨🇳

Translation

Chinese

Agent Eval Harness Ops

Agent Eval Harness 操作规范

Trigger

触发条件

Use when:
  • the user wants to test whether an agent follows instructions or steering docs
  • the user asks for evals, rubrics, failure categories, or test cases for an agent workflow
  • the task involves coding agents, tool-use agents, workflow agents, or skill/AGENTS.md behavior
  • an agent run failed and the user wants a reproducible eval instead of a one-off postmortem
Do not use when:
  • the user only wants a normal code review or prompt rewrite
  • the target behavior is too vague to evaluate
  • the task requires live benchmark infrastructure before a lightweight rubric is useful
适用场景：
  • 用户希望测试Agent是否遵循指令或steering docs
  • 用户询问Agent工作流的评估方法、评分标准、故障分类或测试用例
  • 任务涉及代码Agent、工具使用Agent、工作流Agent，或skill/AGENTS.md中定义的行为
  • Agent运行失败，用户需要可复现的评估而非一次性事后分析
不适用场景：
  • 用户仅需要常规代码审查或提示词改写
  • 目标行为过于模糊无法评估
  • 在轻量级评分标准可用前，任务需要实时基准测试基础设施

Inputs

输入信息

  • Required: agent instructions or steering docs, target task, expected behavior
  • Optional: prior agent transcript, repo/tool constraints, known failure examples, scoring scale, desired output artifact
  • 必填：Agent指令或steering docs、目标任务、预期行为
  • 可选：过往Agent运行记录、仓库/工具约束、已知故障示例、评分尺度、期望输出产物

Workflow

工作流程

  1. Define the evaluation boundary:
    • agent role and allowed tools
    • task type
    • expected final artifact
    • behaviors that must never happen
  2. Convert requirements into behavior categories:
    • instruction adherence
    • tool choice and sequencing
    • preservation of user edits
    • verification and test running
    • recovery from failed commands or missing context
    • output quality and handoff clarity
  3. Build 3-7 focused eval cases:
    • one happy path
    • one ambiguous instruction case
    • one dirty-worktree or preexisting-change case
    • one tool failure or missing-data case
    • one final-report accuracy case
  4. For each case, define:
    • setup/context
    • task prompt
    • expected behaviors
    • unacceptable behaviors
    • pass/fail or 1-5 scoring rubric
  5. If a prior run exists, classify failures against the rubric.
  6. Recommend fixes:
    • steering-doc edits
    • prompt changes
    • tool workflow changes
    • deterministic checks or scripts worth adding
  7. Optionally record an execution receipt when runs need to be compared or audited:
    • Use any installed provider-neutral receipt runner; AI Task Receipt is one example, not a required dependency.
    • Prefer a machine-readable JSON receipt plus a concise human-readable view.
    • Capture task/case id, runner and provider/model when known, permissions, start/end or duration, exit status, bounded/redacted output, verifier result, and residual risk.
    • A receipt records what ran; it does not turn one run into a statistically meaningful benchmark.
    • Do not require paid multi-model runs. Use the smallest run set that can answer the evaluation question, including a single local or existing provider when appropriate.
  1. 定义评估边界：
    • Agent角色与允许使用的工具
    • 任务类型
    • 预期最终产物
    • 绝对禁止的行为
  2. 将需求转化为行为分类：
    • 指令依从性
    • 工具选择与执行顺序
    • 用户编辑内容保留情况
    • 验证与测试运行
    • 从失败命令或缺失上下文恢复的能力
    • 输出质量与交接清晰度
  3. 构建3-7个聚焦的评估用例：
    • 一个正常路径用例
    • 一个指令模糊的用例
    • 一个工作树脏数据或已有变更的用例
    • 一个工具故障或数据缺失的用例
    • 一个最终报告准确性的用例
  4. 为每个用例定义：
    • 前置条件/上下文
    • 任务提示词
    • 预期行为
    • 不可接受行为
    • 通过/失败或1-5分的评分标准
  5. 若存在过往运行记录，对照评分标准对故障进行分类。
  6. 推荐修复方案：
    • steering docs修改
    • 提示词调整
    • 工具工作流变更
    • 值得添加的确定性检查或脚本
  7. 当需要对比或审计运行记录时，可选择性记录执行凭证：
    • 使用已安装的中立供应商凭证运行器；AI Task Receipt是其中一个示例，并非必需依赖。
    • 优先选择机器可读的JSON凭证，辅以简洁的人类可读视图。
    • 捕获任务/用例ID、运行器与供应商/模型（若已知）、权限、开始/结束时间或时长、退出状态、受限/脱敏输出、验证结果以及剩余风险。
    • 凭证仅记录运行内容；它不会将单次运行转化为具有统计意义的基准测试。
    • 无需付费多模型运行。使用能够回答评估问题的最小运行集，包括合适的本地或现有供应商的单次运行。

Output Contract

输出约定

Return:
  • eval objective
  • behavior categories
  • eval cases
  • scoring rubric
  • observed or hypothetical failure taxonomy
  • recommended fixes ranked by leverage
  • next action: run manually, automate, or defer
  • optional receipt path/id and verifier result when a receipt was requested
返回内容：
  • 评估目标
  • 行为分类
  • 评估用例
  • 评分标准
  • 已观测或假设的故障分类体系
  • 按影响力排序的推荐修复方案
  • 下一步行动：手动运行、自动化或延期
  • 若请求凭证，可选凭证路径/ID与验证结果

Boundaries

边界限制

  • Keep evals small enough to run in real work.
  • Separate model capability failures from instruction/design failures.
  • Do not claim statistical benchmark certainty from a tiny harness.
  • Keep receipt tooling optional and provider-neutral; do not make a paid model or a specific runner a prerequisite for lightweight evaluation.
  • Keep examples public-safe: no private paths, credentials, account IDs, or real personal identifiers.
  • 评估规模需小到可在实际工作中运行。
  • 区分模型能力故障与指令/设计故障。
  • 不要声称小型框架能提供统计基准测试的确定性。
  • 保持凭证工具可选且供应商中立；不要将付费模型或特定运行器作为轻量级评估的先决条件。
  • 示例需符合公开安全要求：不得包含私有路径、凭证、账户ID或真实个人身份信息。