Back to Details

harness-engineering

Compare original and translation side by side

🇺🇸

Original

English
🇨🇳

Translation

Chinese

Harness Engineering

Harness Engineering（控制系统工程）

Harness engineering designs the control system around an agent: what it may edit, how it receives feedback, where it writes state, how failures recover, and who can approve irreversible actions. The harness is the difference between a helpful agent session and an autonomous loop that can run for days without corrupting its objective.
Harness Engineering 围绕 Agent 设计控制系统：Agent 可以编辑什么、如何接收反馈、在何处写入状态、失败后如何恢复，以及谁可以批准不可逆操作。Harness 决定了一次有帮助的 Agent 会话，与一个能够连续运行数天且不破坏其目标的自主循环之间的区别。

When to Activate

何时启用

Activate this skill when:
  • Building autonomous research or experimentation loops
  • Designing an agent environment with locked metrics and editable code or content
  • Creating PR-producing or background agents
  • Evaluating whether an agent can safely run without frequent human prompts
  • Adding novelty, ablation, pruning, rollback, or durable logging to an agent workflow
  • Preventing agents from gaming benchmarks, weakening rubrics, or losing state across compaction
Do not activate this skill for adjacent work owned by other skills:
  • General quality gates, regression suites, or outcome metrics without autonomous control surfaces: 
    evaluation
    .
  • Tool schemas, response formats, and recovery errors for harness tools: 
    tool-design
    .
  • Project-level task-model fit, pipeline shape, and cost planning: 
    project-development
    .
  • Remote sandbox, warm-pool, and hosted session infrastructure: 
    hosted-agents
    .
在以下情况下启用此 skill：
  • 构建自主研究或实验循环
  • 设计包含锁定指标和可编辑代码或内容的 Agent 环境
  • 创建能够生成 PR 或在后台运行的 Agent
  • 评估 Agent 是否能在无需频繁人工提示的情况下安全运行
  • 向 Agent workflow 中添加新颖性门控、ablation、剪枝、回滚或持久日志
  • 防止 Agent 钻 benchmark 的空子、弱化 rubric，或在上下文压缩（compaction）后丢失状态
对于由其他 skill 负责的相邻工作，不要启用此 skill：
  • 不具备自主控制 surface 的通用质量门、回归测试套件或结果指标：
    evaluation
  • harness 工具的 tool schema、响应格式和恢复错误：
    tool-design
  • 项目级 task-model 适配、pipeline 结构和成本规划：
    project-development
  • 远程 sandbox、warm pool 和托管会话基础设施：
    hosted-agents

Core Concepts

核心概念

Harness Boundary

Harness Boundary（边界）

Separate the agent from the environment it operates inside. The agent proposes actions; the harness defines allowed surfaces, feedback, persistence, and promotion rules.
Use four surface classes:
SurfaceExamplesRule
LockedEval metric, rubric, validation script, merge policyAgent may read and propose changes, but cannot score itself with modified rules
EditableSkill draft, experiment file, prompt, config under testAgent may mutate during the loop
Append-onlyResults log, research thread, rejected ideasAgent may append, not rewrite
Human-controlledMerge, production deploy, credentials, destructive operationsRequires explicit human approval
将 Agent 与其运行的环境分离。Agent 提出操作；Harness 定义允许的 surface、反馈、持久化和晋级规则。
使用四类 surface：
Surface示例规则
Locked（锁定）评估指标、rubric、验证脚本、合并策略Agent 可以读取并提出更改，但不能使用修改后的规则为自己评分
Editable（可编辑）skill 草稿、实验文件、prompt、测试中的配置Agent 可以在循环期间修改
Append-only（仅追加）结果日志、研究线程、被拒想法Agent 可以追加内容，但不能重写
Human-controlled（人工控制）合并、生产部署、credentials、破坏性操作需要明确的人工批准

Tight Feedback Loops

Tight Feedback Loops（紧密反馈循环）

Autonomy works when feedback is fast, unambiguous, and hard to game. Karpathy's 
autoresearch
is the minimal pattern: one editable file, one locked evaluation file, fixed wall-clock budget, one scalar metric, git rollback, and a durable results log. The lesson is not that every harness needs one metric; it is that ambiguous feedback creates ambiguous autonomy.
For open-ended research-to-skill work, replace the scalar metric with locked rubrics, deterministic structure checks, source traceability, and human review thresholds.
当反馈快速、明确且难以被钻空子时，自主性才能发挥作用。Karpathy 的 
autoresearch
是最小模式：一个可编辑文件、一个锁定的评估文件、固定的时间预算、一个标量指标、git 回滚和持久结果日志。关键启示并不是每个 harness 都需要单一指标，而是模糊的反馈会催生模糊的自主性。
对于开放式 research-to-skill 工作，可以用锁定的 rubric、确定性结构检查、来源可追溯性和人工审查阈值取代标量指标。

Durable State

Durable State（持久状态）

Long-running agents must externalize state. Store plans, source queues, results, failures, and handoffs in files so future agents can resume without relying on chat history. Prime Intellect's autonomous nanoGPT work showed the value of durable scratchpads and 
THREAD.md
-style logs for recovery, monitoring, and audit.
Use append-only logs for:
  • What was tried
  • What improved or failed
  • Why a candidate was kept, discarded, or routed to review
  • Which upstream sources were checked
  • What the next agent should do
长时间运行的 Agent 必须将状态外置。将计划、来源队列、结果、失败和交接记录写入文件，使后续 Agent 无需依赖聊天历史即可恢复。Prime Intellect 的自主 nanoGPT 工作展示了持久 scratchpad 和 
THREAD.md
风格日志在恢复、监控和审计方面的价值。
以下内容应使用 append-only 日志：
  • 尝试过什么
  • 哪些尝试得到改善或失败
  • 为什么保留、丢弃候选方案或将其转交审查
  • 检查了哪些上游来源
  • 下一个 Agent 应该做什么

Search Discipline

Search Discipline（搜索纪律）

Agents tend to exploit the nearest surface, stack complexity, and under-run pruning. Add explicit search rules:
  1. Refresh upstream sources on a schedule.
  2. Require novelty checks before spending large budgets.
  3. Preserve rejected attempts to avoid rediscovery.
  4. Run leave-one-out pruning when a stack has multiple additions.
  5. Reward simplification when quality is equal.
  6. Use separate verification before promotion.
Agent 往往会利用最容易触及的 surface、堆积复杂性，并让剪枝执行不足。应添加明确的搜索规则：
  1. 按计划刷新上游来源。
  2. 在投入大量预算前，要求进行新颖性检查。
  3. 保留被拒尝试，避免重复发现。
  4. 当一个 stack 中包含多项新增内容时，运行 leave-one-out pruning。
  5. 质量相同时，奖励简化。
  6. 晋级前使用独立验证。

Mechanism Registry

Mechanism Registry（机制注册表）

For research-to-skill systems, track accepted mechanisms separately from prose. A mechanism record should include a stable 
mechanism_id
, 
owning_skill
, 
status
, activation scenario, behavior change, evidence, and failure modes. Novelty gates should compare against this registry before using broader corpus overlap, because keyword overlap catches stale phrasing while mechanism comparison catches real duplication.
对于 research-to-skill 系统，应将已接受的 mechanism 与说明性文字分开跟踪。mechanism 记录应包含稳定的 
mechanism_id
、owning_skill、status、激活场景、行为变化、证据和失败模式。新颖性门控应先与该 registry 进行比较，再使用更广泛的语料重叠检查，因为关键词重叠能够发现陈旧措辞，而 mechanism 比较才能发现真正的重复。

Governance

治理

Autonomous agents may prepare PRs, but governance must be explicit. They can draft changes, run checks, and write PR summaries. They should not merge, deploy, or push without human approval unless the user has explicitly granted that permission for the specific action.
自主 Agent 可以准备 PR，但治理必须明确。它们可以起草更改、运行检查并编写 PR 摘要。除非用户已明确授予特定操作的权限，否则不应在未经人工批准的情况下执行 merge、deploy 或 push。

Detailed Topics

详细主题

Autoresearch-Style Loop

Autoresearch 风格循环

Use this pattern when optimizing an artifact against a stable evaluator:
text
read locked context -> choose hypothesis -> edit allowed surface -> commit/checkpoint
-> run evaluator -> log result -> keep if better -> discard or rollback if worse
-> repeat
Required properties:
  • The evaluator is outside the editable surface.
  • The feedback cadence is fixed enough to compare attempts.
  • Failed attempts leave an audit trail.
  • Rollback is cheap.
  • The agent has a policy for crashes and timeouts.
在针对稳定 evaluator 优化 artifact 时，使用此模式：
text
read locked context -> choose hypothesis -> edit allowed surface -> commit/checkpoint
-> run evaluator -> log result -> keep if better -> discard or rollback if worse
-> repeat
必需属性：
  • evaluator 位于可编辑 surface 之外。
  • 反馈节奏足够固定，以便比较各次尝试。
  • 失败尝试会留下审计轨迹。
  • 回滚成本低。
  • Agent 具备针对崩溃和超时的策略。

Research-To-Skill Loop

Research-To-Skill 循环

Use this pattern when sources become skill changes:
text
discover -> retrieve -> gate -> score -> extract mechanism
-> map to existing or new skill -> draft proposal -> validate structure
-> prepare PR -> human review
The locked evaluator is a combination of source rubrics, skill-change rubrics, structure checks, and reviewer approval. The editable artifact is the proposed skill delta.
当来源转化为 skill 更改时，使用此模式：
text
discover -> retrieve -> gate -> score -> extract mechanism
-> map to existing or new skill -> draft proposal -> validate structure
-> prepare PR -> human review
锁定的 evaluator 由来源 rubric、skill 变更 rubric、结构检查和审查者批准共同组成。可编辑的 artifact 是拟议的 skill 增量。

Metric Gaming Resistance

抵御指标博弈

Assume an optimizing agent will learn the harness. Guard against:
  • Editing evaluation code or rubrics and then using the new version for self-approval
  • Adding verbose content that pleases a judge but harms skill activation
  • Citing unretrieved sources
  • Optimizing aggregate scores while failing a critical dimension
  • Avoiding failed results in the log
Mitigation: lock rubrics per run, report per-dimension scores, require source retrieval evidence, preserve rejected attempts, and route governance changes to human review.
假设优化型 Agent 会逐渐适应并利用 harness。需防范以下情况：
  • 修改评估代码或 rubric，然后使用修改后的版本自行批准
  • 添加冗长内容以取悦评判者，却损害 skill 激活
  • 引用未实际检索的来源
  • 优化汇总分数，却未能通过某个关键维度
  • 在日志中回避失败结果
缓解措施：每次运行锁定 rubric，报告各维度分数，要求提供来源检索证据，保留被拒尝试，并将治理变更交由人工审查。

Monitoring Agents

监控 Agent

Use monitoring agents for long runs, but restrict them to read-only reporting unless explicitly tasked otherwise. Monitoring output should report:
  • Best current candidate
  • Active jobs or drafts
  • Last upstream refresh
  • Failed or stale loops
  • Disagreements between logs and claimed state
  • Next action and blocker
长时间运行期间可以使用监控 Agent，但除非明确分配了其他任务，否则应将其限制为只读报告。监控输出应报告：
  • 当前最佳候选方案
  • 活跃任务或草稿
  • 最近一次上游刷新
  • 失败或过期的循环
  • 日志与声明状态之间的不一致
  • 下一步操作和阻塞项

Practical Guidance

实践指南

Harness Design Checklist

Harness 设计检查清单

  1. Define the objective in one sentence.
  2. Identify locked, editable, append-only, and human-controlled surfaces.
  3. Choose the feedback mechanism: scalar metric, rubric, deterministic tests, human review, or combination.
  4. Define keep, discard, crash, timeout, and review states.
  5. Create a durable thread log before the loop starts.
  6. Add source refresh, mechanism-registry novelty, and pruning rules for long-running loops.
  7. Define what the agent may do without asking and what requires approval.
  8. Validate the harness on one known good and one known bad artifact.
  1. 用一句话定义目标。
  2. 确定 Locked、Editable、Append-only 和 Human-controlled surface。
  3. 选择反馈机制：标量指标、rubric、确定性测试、人工审查或其组合。
  4. 定义保留、丢弃、崩溃、超时和审查状态。
  5. 在循环开始前创建持久 thread log。
  6. 为长时间运行的循环添加来源刷新、mechanism registry 新颖性检查和剪枝规则。
  7. 定义 Agent 可以不经询问执行的操作，以及必须获得批准的操作。
  8. 使用一个已知良好的 artifact 和一个已知有问题的 artifact 验证 harness。

File Layout

文件布局

text
research-run/
  THREAD.md
  sources/
    queue.md
    evaluations/
  proposals/
  logs/
    results.tsv
    rejected.md
  drafts/
Use TSV or JSONL for append-only machine-readable logs. Use Markdown for handoffs and reviewer-facing summaries.
text
research-run/
  THREAD.md
  sources/
    queue.md
    evaluations/
  proposals/
  logs/
    results.tsv
    rejected.md
  drafts/
仅追加的机器可读日志应使用 TSV 或 JSONL。交接记录和面向审查者的摘要应使用 Markdown。

Examples

示例

Example 1: Locked metric
An agent optimizes 
train.py
, but 
prepare.py
owns data loading and evaluation. The agent can edit the model but cannot change the metric. Failed experiments are logged and rolled back.
Example 2: Locked rubric
An agent evaluates a new Anthropic or OpenAI engineering post, but the source curation rubric is locked for the run. If the source passes, the agent drafts a skill proposal. It cannot lower the rubric threshold to admit the source.
Example 3: Auto-PR without auto-merge
An agent prepares a branch and PR body after passing source, skill, and structure checks. The PR states unresolved risks and waits for human merge approval.
示例 1：锁定指标
Agent 优化 
train.py
，但由 
prepare.py
负责数据加载和评估。Agent 可以编辑模型，却不能更改指标。失败的实验会被记录并回滚。
示例 2：锁定 rubric
Agent 评估一篇新的 Anthropic 或 OpenAI 工程文章，但本次运行期间，来源筛选 rubric 处于锁定状态。如果来源通过检查，Agent 会起草一份 skill 提案。它不能降低 rubric 阈值来接纳该来源。
示例 3：自动准备 PR，但不自动合并
Agent 在通过来源、skill 和结构检查后准备分支和 PR 正文。PR 会说明尚未解决的风险，并等待人工合并审批。

Guidelines

准则

  1. Lock evaluators before starting the loop.
  2. Keep editable surfaces narrow enough for reliable diffs.
  3. Write durable logs before context compaction can erase state.
  4. Report per-dimension scores instead of only aggregate scores.
  5. Require source retrieval before citation.
  6. Add novelty gates for broad search and pruning gates for complex stacks.
  7. Prefer simplification when quality is equal.
  8. Separate PR preparation from merge authority.
  9. Revalidate harness changes with old and new evaluators.
  10. Treat stopped autonomous loops as harness failures, not agent personality quirks.
  1. 在循环开始前锁定 evaluator。
  2. 将 Editable surface 限制得足够窄，以确保 diff 可靠。
  3. 在上下文压缩可能抹除状态前写入持久日志。
  4. 报告各维度分数，而不只是汇总分数。
  5. 引用前必须实际检索来源。
  6. 为广泛搜索添加 novelty gate，为复杂 stack 添加 pruning gate。
  7. 质量相同时优先选择简化方案。
  8. 将 PR 准备与 merge 权限分离。
  9. 使用新旧 evaluator 重新验证 harness 变更。
  10. 将停止的自主循环视为 harness 故障，而不是 Agent 的个性怪癖。

Gotchas

易错点

  1. Mutable evaluator: If the agent can edit the metric, it may optimize the benchmark instead of the task. Keep rubrics and eval code locked during the run.
  2. Chat-only memory: Long runs fail after compaction when plans live only in conversation history. Write thread logs and result files from the start.
  3. No discard record: Without rejected-attempt logs, agents repeat failed ideas. Preserve failures with enough detail to avoid rediscovery.
  4. Complexity accretion: Agents stack changes and rarely remove them. Require pruning rounds and reward equal-quality simplification.
  5. Premature novelty claims: Agents label recombinations as novel. Compare against existing repo skills, source queue, and rejected logs before claiming novelty.
  6. Monitor misreporting: Monitoring agents can summarize stale or inconsistent state. Require them to cite the files or logs behind claims.
  7. Human approval ambiguity: "Prepare a PR" is not "merge a PR." Make approval boundaries explicit in the harness.
  8. Volatile source drift: Fast-moving lab claims age quickly. Put dated evidence in references and schedule revalidation.
  1. 可变 evaluator：如果 Agent 能编辑指标，它可能优化 benchmark，而不是实际任务。在运行期间保持 rubric 和评估代码锁定。
  2. 仅聊天记忆：当计划只存在于对话历史中时，长时间运行会在上下文压缩后失败。从一开始就应写入 thread log 和结果文件。
  3. 无丢弃记录：没有 rejected-attempt 日志，Agent 会重复尝试失败的想法。应保留失败及足够细节，避免重复发现。
  4. 复杂性累积：Agent 不断叠加变更，却很少删除。强制执行 pruning 轮次，并奖励质量相同的简化。
  5. 过早声称新颖性：Agent 会将重组视为 novel。声称新颖前，应与现有 repo skill、来源队列和 rejected 日志进行比较。
  6. 监控误报：监控 Agent 可能总结过时或不一致的状态。要求其引用支持结论的文件或日志。
  7. 人工审批歧义：“准备 PR”不等于“合并 PR”。应在 harness 中明确审批边界。
  8. 来源快速漂移：快速变化的实验室声明很快就会过时。在 references 中记录带日期的证据，并安排重新验证。

Integration

集成

This skill connects to:
  • evaluation - Rubrics and quality gates provide the locked feedback surface
  • advanced-evaluation - Pairwise comparison and bias mitigation improve proposal review
  • filesystem-context - Durable logs, scratchpads, and thread files preserve state
  • multi-agent-patterns - Researcher, verifier, monitor, and writer agents need isolated contexts
  • tool-design - Harness tools must expose clear contracts and recovery errors
  • project-development - File-based pipelines and task-model fit analysis keep loops simple
  • hosted-agents - Background execution needs sandbox, snapshot, and approval boundaries
此 skill 与以下 skill 相连：
  • evaluation
    - rubric 和质量门提供锁定的反馈 surface
  • advanced-evaluation
    - 成对比较和偏差缓解可改进提案审查
  • filesystem-context
    - 持久日志、scratchpad 和 thread 文件保存状态
  • multi-agent-patterns
    - researcher、verifier、monitor 和 writer Agent 需要隔离上下文
  • tool-design
    - harness 工具必须提供清晰的契约和恢复错误
  • project-development
    - 基于文件的 pipeline 和 task-model 适配分析可保持循环简单
  • hosted-agents
    - 后台执行需要 sandbox、snapshot 和审批边界

References

参考资料

Internal references:
  • researcher/README.md
    - Read when implementing the repo-native research-to-skill operating system
  • researcher/rubrics/harness-change.md
    - Read when evaluating changes to an agent harness
  • researcher/runbooks/autonomous-research-loop.md
    - Read when running a source-to-skill loop
External resources:
  • Karpathy 
    autoresearch
    - Constrained autonomous experiment loop with locked evaluation
  • Prime Intellect autonomous nanoGPT speedrun - Durable scratchpads, handoffs, monitoring, and autonomy failure modes
  • AlphaEvolve and FunSearch - LLM-generated candidates paired with systematic evaluators
  • HELM and LM Evaluation Harness - Transparent, reproducible evaluation infrastructure
内部参考资料：
  • researcher/README.md
    - 实现仓库原生的 research-to-skill 操作系统时阅读
  • researcher/rubrics/harness-change.md
    - 评估 Agent harness 变更时阅读
  • researcher/runbooks/autonomous-research-loop.md
    - 运行 source-to-skill 循环时阅读
外部资源：
  • Karpathy 
    autoresearch
    - 带锁定评估的受限自主实验循环
  • Prime Intellect autonomous nanoGPT speedrun - 持久 scratchpad、交接、监控和自主性故障模式
  • AlphaEvolve 和 FunSearch - LLM 生成的候选方案与系统化 evaluator 配对
  • HELM 和 LM Evaluation Harness - 透明、可复现的评估基础设施

Skill Metadata

Skill 元数据

Created: 2026-05-14 Last Updated: 2026-05-15 Author: Agent Skills for Context Engineering Contributors Version: 1.1.0
创建时间：2026-05-14 最后更新时间：2026-05-15 作者：Agent Skills for Context Engineering Contributors 版本：1.1.0