Back to Details

eval-engineering

Compare original and translation side by side

🇺🇸

Original

English
🇨🇳

Translation

Chinese

Eval Engineering

评估工程

Build evals iteratively:
text
inspect agent and interview user -> propose directions -> user chooses
-> approve runtime and environment -> build, run, audit -> review and repeat
Use the latest version of Harbor. Put task source under 
evals/
. Read references/harbor.md before creating or running a task.
迭代式构建评估：
text
检查Agent并与用户沟通 -> 提出方向 -> 用户选择
-> 批准运行时与环境 -> 构建、运行、审核 -> 评审并重复
使用最新版本的Harbor。将任务源放置在
evals/
目录下。在创建或运行任务前，请阅读references/harbor.md。

1. Map the agent

1. 映射Agent

Inspect the active agent and code reachable from its public entrypoint. Find:
  • runtime: entrypoint, input/output, prompts, models, routing, retries, hooks, middleware, and memory;
  • actions: tools, inputs, outputs, failures, external dependencies, and effects;
  • backing data: documents, records, indexes, files, policies, schemas, and source/version when available;
  • state: identity, permissions, filesystem, network, time, sessions, and mutable state;
  • purpose: intended users, jobs, and what a good result provides;
  • evidence: tests, fixtures, issues, existing evals, and documented failures.
Mapping is read-only. Do not start the target or services, install packages, or use external credentials before the user approves the runtime and environment.
Summarize the map in the conversation:
text
Agent: target and entrypoint
Purpose: users and jobs
Abilities: work it is expected to perform
Tools and data: actions, backing data, and dependencies
Effects: reads, writes, and state changes
Evidence: tests, failures, or traces
Use code to model the agent; do not turn implementation details into the eval question unless answering questions about that code is the agent's job.
Keep the user involved: explain the map and what it implies in plain language, then ask only for information the repository and traces cannot establish. For example: “Which user job matters most?”, “What failure should never happen?”, or “What would a good result look like?”
检查活跃Agent及其公开入口可访问的代码。找出以下内容：
  • 运行时：入口点、输入/输出、提示词、模型、路由、重试机制、钩子、中间件和内存；
  • 操作：工具、输入、输出、失败情况、外部依赖和影响；
  • 支撑数据：文档、记录、索引、文件、策略、模式，以及可用的来源/版本；
  • 状态：身份、权限、文件系统、网络、时间、会话和可变状态；
  • 用途：目标用户、任务，以及优质结果能提供的价值；
  • 证据：测试用例、测试数据、问题记录、现有评估和已记录的失败案例。
映射过程为只读操作。在用户批准运行时和环境之前，请勿启动目标Agent或相关服务、安装包或使用外部凭据。
在对话中总结映射结果：
text
Agent：目标及入口点
用途：用户及任务
能力：预期执行的工作
工具与数据：操作、支撑数据及依赖
影响：读取、写入及状态变更
证据：测试、失败案例或追踪数据
使用代码建模Agent；除非Agent的工作是回答关于该代码的问题，否则请勿将实现细节转化为评估问题。
让用户全程参与：用通俗易懂的语言解释映射结果及其含义，然后仅询问仓库和追踪数据无法确定的信息。例如：“哪项用户任务最为重要？”、“哪种绝对不能发生的失败情况？”或“优质结果应该是什么样的？”

Optional traces

可选追踪数据

Use traces only when the user provides a source or asks to use them. Read references/trace-sourcing.md. Use selected traces to identify real requests and dependency behavior, and never treat a recorded target answer as truth.
仅当用户提供来源或要求使用时，才使用追踪数据。请阅读references/trace-sourcing.md。使用选定的追踪数据识别真实请求和依赖行为，但切勿将记录的目标Agent答案视为事实。

2. Discuss and choose an eval direction

2. 讨论并选择评估方向

Propose two or three capabilities grounded in the map. For each, give:
text
Name
Example request: realistic request sent to the agent
Tests: behavior the eval distinguishes
Needs: main obstacle, data, or environment requirement
Example:
text
Name: choose the right account lookup
Example request: “What plan is account A on?”
Tests: retrieves account A, uses the returned plan, and does not invent account details
Needs: a read-only account lookup with known records
Recommend one and ask the user which to build. The request must make the agent exercise the capability: use multiple turns for context use, competing tools for tool choice, source material for retrieval, or known state for an action.
Do not implement until the user chooses.
基于映射结果提出两到三个能力方向。每个方向需包含：
text
名称
示例请求：发送给Agent的真实请求
测试点：评估需区分的行为
需求：主要障碍、数据或环境要求
示例：
text
名称：选择正确的账户查询方式
示例请求：“账户A使用的是什么套餐？”
测试点：检索账户A、使用返回的套餐信息、不编造账户细节
需求：带有已知记录的只读账户查询
推荐一个方向并询问用户要构建哪一个。请求必须让Agent发挥对应能力：使用多轮对话来测试上下文运用、竞争工具来测试工具选择、源材料来测试检索能力，或已知状态来测试操作执行。
在用户做出选择前，请勿开始实现。

3. Checkpoint: approve runtime and environment

3. 检查点：批准运行时与环境

Read references/task-design.md and references/environment-building.md. Design one scenario that requires the selected capability.
Recommend a target runtime:
  • Active entrypoint: preserve the repository's agent behavior. Recommend this when it can run safely.
  • Reconstruction: use only when the active entrypoint cannot run in a controlled eval. Name the behavior it cannot preserve and label the eval as a reconstruction.
Before implementation, give the user one proposal under 150 words:
text
Task: request and capability
Runtime: active entrypoint or reconstruction, with tradeoff
Dependencies and backing data: live, frozen, or simulated; required credentials if live, effects, and source/version
Success: how the result is judged
Recommendation: preferred setup and why
The user approves or revises the target runtime and environment boundary. Never write to production; isolate mutations.
阅读references/task-design.md和references/environment-building.md。设计一个需要运用选定能力的场景。
推荐目标运行时：
  • 活跃入口点： 保留仓库中Agent的原有行为。当Agent可安全运行时，推荐此选项。
  • 重构： 仅当活跃入口点无法在受控评估中运行时使用。说明其无法保留的行为，并将评估标记为重构类型。
在实现前，向用户提交一份不超过150字的提案：
text
任务：请求及对应的能力
运行时：活跃入口点或重构，以及权衡点
依赖与支撑数据：实时、冻结或模拟；若为实时则需提供必要凭据、影响及来源/版本
成功标准：结果的评判方式
推荐方案：首选设置及其原因
用户将批准或修改目标运行时和环境边界。切勿写入生产环境；隔离所有变更操作。

4. Build one Harbor task

4. 构建一个Harbor任务

Create one task for the selected capability:
text
evals/<task-id>/
├── task.toml
├── instruction.md
├── environment/
└── tests/
Add an adapter or non-default configuration only when Harbor needs it to invoke the approved target runtime. Keep instructions and environment facts visible to the target; keep expected outcomes, judge criteria, and judge credentials unavailable to it.
An adapter may translate I/O and inject approved dependencies. It must not make target decisions, contain answers, or fabricate actions. Custom adapters and verifiers must write the target response/action record, verifier evidence, verdict/reason, reward, and errors to Harbor artifacts or verifier logs. Do not add 
audit.json
or another result ledger.
Use an LLM judge for semantic success and deterministic checks for execution, parsing, files, or state. Read references/verifier-design.md. Emit one primary reward.
为选定的能力创建一个任务：
text
evals/<task-id>/
├── task.toml
├── instruction.md
├── environment/
└── tests/
仅当Harbor需要调用已批准的目标运行时才添加适配器或非默认配置。让指令和环境事实对目标Agent可见；将预期结果、评判标准和评判凭据对目标Agent隐藏。
适配器可转换输入输出并注入已批准的依赖。但不得替目标Agent做决策、包含答案或伪造操作。自定义适配器和验证器必须将目标响应/操作记录、验证器证据、verdict/reason、奖励和错误写入Harbor工件或验证器日志。请勿添加
audit.json
或其他结果账本。
使用LLM评判语义层面的成功，使用确定性检查来验证执行、解析、文件或状态相关的内容。请阅读references/verifier-design.md。输出一个主要奖励值。

5. Test, run, and audit

5. 测试、运行并审核

Start the minimum environment before completing the scenario. Test the verifier directly with one clearly valid result that passes and one realistic incorrect result that fails.
Run the approved target runtime through Harbor. Inspect:
  • target response and trajectory;
  • harness-observed tool calls, actions, and state;
  • verifier evidence, verdict, reason, reward, and errors;
  • resolved target and environment configuration.
Fix and rerun when the task is unclear, the environment is unrealistic, the verifier is wrong, or infrastructure failed. Before approval, confirm that the target exercised the selected capability and the verifier scored that behavior, not an environment or verifier failure. If the environment returned the answer before the target used the intended tool, revise the task.
在完成场景设置前启动最小化环境。直接测试验证器：用一个明显有效的结果测试通过情况，用一个真实的错误结果测试失败情况。
通过Harbor运行已批准的目标运行时。检查以下内容：
  • 目标Agent的响应和执行轨迹；
  • 测试框架观测到的工具调用、操作和状态；
  • 验证器的证据、verdict/reason、奖励和错误；
  • 已解析的目标Agent和环境配置。
当任务描述不清晰、环境不真实、验证器判断错误或基础设施故障时，修复并重新运行。在批准前，确认目标Agent已运用选定的能力，且验证器针对该行为打分，而非针对环境或验证器故障打分。如果环境在目标Agent使用预期工具前就返回了答案，请修改任务。

6. Review with the user

6. 与用户评审

Explain the task path and run command, capability and scenario, runtime and dependency boundary, target behavior, verifier decision, and limitation. Ask the user to approve, revise, drop, or choose the next direction. If continuing, reuse the map and trace findings, then propose a distinct capability.
解释任务路径和运行命令、能力与场景、运行时与依赖边界、目标Agent行为、验证器决策及局限性。询问用户是批准、修改、放弃还是选择下一个方向。如果继续，复用映射和追踪数据的发现，然后提出一个不同的能力方向。

Invariants

不变规则

  • One capability per Harbor task under 
    evals/
    .
  • No production writes; reset mutable state between trials.
  • Keep hidden truth and judge credentials unavailable to the target.
  • Treat build, credential, reset, timeout, judge, and verifier failures as infrastructure errors.
  • 每个Harbor任务对应一个能力，放置在
    evals/
    目录下。
  • 禁止写入生产环境；在测试轮次之间重置可变状态。
  • 对目标Agent隐藏真实情况和评判凭据。
  • 将构建、凭据、重置、超时、评判和验证器故障视为基础设施错误。