Harness Creator

Harness Creator（中文版）

Choose Your Problem

选择你要解决的问题

The Five-Subsystem Harness Framework

1. Instructions (Recipe Shelf): AGENTS.md, CLAUDE.md, docs/ hierarchy
2. State (Prep Station): feature_list.json, progress.md, session-handoff.md
3. Verification (Quality Check Window): Verification commands, test suites, type checks
4. Scope (Task Boundaries): One-feature-at-a-time policies, definition of done
5. Lifecycle (Session Management): init.sh, clean-state checklists, handoff procedures

Creating a Harness

Phase 1: Context Gathering

1. What project is this for? (tech stack, size, complexity)
2. What agent tool are they using? (Claude Code, Codex, Cursor, etc.)
3. What exists already? (any AGENTS.md, progress tracking, verification?)
4. What problems are they experiencing? (agent overreach, lost context, broken tests?)
5. What's the team's tolerance for structure? (minimal vs. comprehensive)

Phase 2: Harness Assessment (Existing Projects)

• 5: Exemplary, documented, consistently followed
• 4: Good, mostly complete, occasional gaps
• 3: Adequate, covers basics, missing polish
• 2: Weak, incomplete, inconsistently applied
• 1: Missing or actively harmful

Phase 3: Design

• Create a short AGENTS.md (~50-100 lines) as the routing layer
• Link to detailed docs in docs/ directory (ARCHITECTURE.md, PRODUCT.md, etc.)
• Define startup workflow: what the agent reads before coding

• Create feature_list.json with feature definitions and status tracking
• Create or update progress.md for session continuity
• Design session-handoff.md template if needed

• List explicit verification commands in AGENTS.md
• Ensure init.sh runs verification
• Design quality score tracking if appropriate

• Define one-feature-at-a-time policy
• Document feature dependencies
• Create definition of done checklist

• Create init.sh for initialization
• Design clean-state checklist
• Document session handoff procedure

Phase 4: Implementation

# Use bundled scripts from scripts/ directory
# (See scripts/ section for available tools)

Phase 5: Testing and Benchmarking

1. Baseline: Run a representative task without the harness
2. With Harness: Run the same task with the harness
3. Measure: Success rate, time, token usage, rework
4. Compare: Quantify the improvement

Harness File Templates

AGENTS.md Structure

# AGENTS.md

[One-sentence project purpose]

## Startup Workflow

Before writing code:
1. [Step 1: e.g., Read this file]
2. [Step 2: e.g., Read ARCHITECTURE.md]
3. [Step 3: e.g., Run ./init.sh]
4. [Step 4: e.g., Read feature_list.json]

## Working Rules

- [Rule 1: e.g., One feature at a time]
- [Rule 2: e.g., Verification required before claiming done]
- [Rule 3: e.g., Update progress before ending session]

## Required Artifacts

- `feature_list.json`: Feature state tracker
- `progress.md`: Session continuity log
- `init.sh`: Standard startup and verification

## Definition of Done

A feature is done when:
- [ ] Implementation complete
- [ ] Verification passed
- [ ] Evidence recorded
- [ ] Repository restartable

## End of Session

Before ending:
1. Update progress.md
2. Update feature_list.json
3. Record blockers/risks
4. Commit with descriptive message
5. Leave clean restart path

feature_list.json Structure

{
  "features": [
    {
      "id": "feat-001",
      "name": "Document Import",
      "description": "Allow users to import PDF and TXT documents",
      "dependencies": [],
      "status": "done",
      "evidence": "tests pass, manual verification on 2024-01-15"
    },
    {
      "id": "feat-002",
      "name": "Document Chunking",
      "description": "Split documents into ~500 char chunks with metadata",
      "dependencies": ["feat-001"],
      "status": "in-progress",
      "evidence": ""
    }
  ]
}

init.sh Structure

#!/bin/bash
set -e

echo "=== Installing dependencies ==="
npm install

echo "=== Running type check ==="
npm run check

echo "=== Running tests ==="
npm test

echo "=== Building application ==="
npm run build

echo "=== Verification complete ==="

Running Benchmarks

Step 1: Define Representative Tasks

• Real work the user would actually do
• Challenging enough to fail without proper harness
• Verifiable (clear success criteria)

Step 2: Run Comparative Sessions

• Without Harness: Run the task on a clean repo copy
• With Harness: Run the same task with the harness in place

• Success/failure
• Time taken
• Token usage
• Rework required
• Session restarts needed

Step 3: Aggregate Results

• Success rate improvement
• Time efficiency change
• Token efficiency change
• Qualitative feedback

Step 4: Iterate

• Which harness components add most value
• Which components are over-engineered
• Where to focus improvement efforts

Bundled Resources

References (Deep-Dive Patterns)

Templates

• templates/agents.md

  — AGENTS.md / CLAUDE.md skeleton

• templates/feature-list.json

  — Feature state tracker

• templates/init.sh

  — Standard initialization script

• templates/progress.md

  — Session progress log

• templates/session-handoff.md

  — Session handoff template

Scripts (Optional)

• scripts/create-harness.ts

  — Generate harness files from templates

• scripts/validate-harness.ts

  — Check harness completeness

• scripts/run-benchmark.ts

  — Execute harness effectiveness comparison

Gotchas

1. Memory index caps fire silently — Long entries invisible once cap hit. Keep hooks to one line.
2. Priority ordering counterintuitive — Local beats project beats user beats org. Test full stack.
3. Extraction timing creates race window — User can start next turn before background extraction completes.
4. Derivable content doesn't belong in memory — Architecture and code patterns are in the repo already.
5. Concurrent classification is per-call, not per-tool — Same tool safe for some inputs, unsafe for others.
6. Permission evaluation has side effects — Tracks denials, transforms modes, updates state.
7. Most async work skips "pending" state — Work units register directly as "running".
8. Fork children must not fork — Recursive guard preserves single-level invariant.
9. Context builders memoized but manually invalidated — Add invalidation or face staleness.
10. Hook trust all-or-nothing — One untrusted hook disables entire extension system.
11. Eviction requires notification — Terminal work unit only GC-eligible after parent notified.
12. Skill listing budgets tight — Front-load distinctive trigger language, tails get cut.

陷阱（Gotchas）

1. 记忆索引上限静默触发 — 条目过长超上限后不可见。保持钩子单行。
2. 优先级顺序反直觉 — 本地胜过项目胜过用户胜过组织。测试完整栈。
3. 提取时序产生竞争窗口 — 用户可在后台提取完成前开始下一轮。
4. 可推导内容不应存入记忆 — 架构和代码模式已在仓库中。
5. 并发分类按调用而非按工具 — 同一工具对某些输入安全，对其他不安全。
6. 权限评估有副作用 — 跟踪拒绝、转换模式、更新状态。
7. 大多数异步工作跳过"pending"状态 — 工作单元直接注册为"运行中"。
8. Fork 子节点不能 Fork — 递归防护保持单层不变量。
9. 上下文构建器缓存但手动失效 — 添加失效或面对过时。
10. Hook 信任全有或全无 — 一个不可信 hook 禁用整个扩展系统。
11. 驱逐需要通知 — 终端工作单元仅在父节点通知后可 GC。
12. Skill 列表预算紧张 — 前置独特触发语言，尾部被截断。

When to Use This Skill

• User says "I need to set up AGENTS.md for my project"
• User wants to improve their agent's reliability
• User is experiencing agent failures, lost context, or broken work
• User asks "how do I make my agent work better?"
• User wants to benchmark harness effectiveness
• User needs templates for harness files
• User is following the Learn Harness Engineering course

Communication Style

• Explain harness concepts in practical terms (kitchen analogy works well)
• Focus on measurable outcomes, not theoretical perfection
• Start minimal, add structure as needed
• Show before/after comparisons to build confidence
• Acknowledge tradeoffs (more structure = more reliability but more upfront work)

Getting Started

If the user is new to harness engineering:

1. Start with assessment: Run the five-tuple assessment on their current setup
2. Pick lowest-scoring subsystem: Focus improvement efforts there first
3. Create minimal viable harness: AGENTS.md + init.sh + feature_list.json
4. Test with real task: Measure before/after improvement

If the user is experienced:

1. Ask what specific problem: Don't assume — let them describe the pain point
2. Understand harness maturity: What exists already? What's working?
3. Design targeted improvements: Use reference patterns for guidance
4. Optionally run benchmarks: Quantify impact with before/after comparison

When NOT to Use This Skill

• Prompt engineering or system prompt design
• Model selection or fine-tuning
• Generic software architecture (MVC, microservices)
• Chat UIs or conversational interfaces
• LLM API integration basics

Further Resources

• Learn Harness Engineering Documentation
• OpenAI: Harness Engineering
• Anthropic: Effective Harnesses for Long-Running Agents
• Anthropic: Harness Design for Long-Running Apps
• Awesome Harness Engineering
• Agentic Harness Patterns Skill — Reference implementation for pattern extraction

Further Resources

• Learn Harness Engineering Documentation
• OpenAI: Harness Engineering
• Anthropic: Effective Harnesses for Long-Running Agents
• Awesome Harness Engineering