ablate-ai-layer

Compare original and translation side by side

🇺🇸

Original

English

🇨🇳

Translation

Chinese

Ablate the AI layer

AI指令层消融测试

Model upgrades quietly retire instructions. A rule written to work around a weaker model becomes dead weight that competes for attention with the rules that still matter. Reading the file will not tell you which is which. Only an experiment will.

You run the experiment. The user picks the task and approves the conclusion. Do not hand the user a list of commands to run; the script drives both arms.

模型升级会悄然让旧指令失效。为适配弱模型而编写的规则会变成无效负担，与仍有用的规则争夺模型注意力。仅通过阅读指令文件无法区分有效与无效规则，唯有实验才能得出结论。

由你执行实验，用户负责选择任务并确认结论。 不要给用户提供命令列表让他们自行执行；脚本会自动完成两组测试的全部流程。

What makes the result trustworthy

为何测试结果值得信赖

Both arms, many runs each. A stripped agent does not visibly fail, so a single run has nothing to compare against and "seems fine" becomes "delete something load-bearing". Two runs of the same arm can also differ more than the two arms differ, so one pair per arm is the floor, not the target.
Nothing is moved aside. Every run happens in a detached git worktree built from HEAD in a temp directory, outside the repo, and deleted afterwards. The user's working tree is never modified, so there is no restore step to forget.
Only the always-loaded set is stripped by default. Skills, subagents and path-scoped rules cost nothing until they fire, so deleting them buys back no context. Hooks and permissions are never touched: they run as code and spend no attention.

两组测试，每组多次运行。移除指令层的Agent不会出现明显故障，因此单次运行没有可对比的基准，“看起来没问题”可能会导致“删除了关键规则”的后果。同一组内的两次运行差异也可能大于两组之间的差异，因此每组至少运行两次是最低要求，而非目标次数。
无需移动任何文件。每次运行都会在仓库外的临时目录中基于HEAD创建独立的git worktree，测试完成后立即删除。用户的工作树绝不会被修改，因此无需担心忘记恢复步骤。
默认仅移除始终加载的指令集。Skill、子Agent和路径范围规则在触发前不会占用任何资源，因此删除它们无法节省上下文资源。钩子（Hooks）和权限设置永远不会被改动：它们以代码形式运行，不会占用模型注意力。

Step 1. Map the layer

步骤1：梳理AI指令层

bash

python <skill>/scripts/map_layer.py [repo_root]

Read-only. Sorts every artifact into always-loaded, on-demand, and enforcement, and prints what the always-loaded set costs on every session before the user types anything. Show them that number.

If nothing is found, say so and stop. There is nothing to test.

bash

python <skill>/scripts/map_layer.py [repo_root]

该脚本为只读模式。它会将所有AI指令 artifact 分类为“始终加载”“按需加载”和“强制执行”三类，并在用户执行任何操作前，显示始终加载的指令集在每次会话中占用的资源成本。请将该数值展示给用户。

如果未找到任何AI指令，需告知用户并停止流程——没有可测试的内容。

Step 2. Get the probe task

步骤2：确定测试任务

This is the one thing you must not decide for the user. The task determines whether the experiment can detect anything at all.

A good probe task is real work they would do anyway, touches code where house conventions plausibly apply, and adds something that has to be wired in: a test, an endpoint, a migration, a command.

A bad one is a typo, a rename, or any one-line fix. It is fully derivable, both arms will match, and the user will wrongly conclude their whole layer is worthless. Say that out loud if they offer one, and ask for something with conventions at stake.

Write the agreed task verbatim to a file. Every run reuses it byte for byte.

这是唯一不能替用户决定的事项。测试任务决定了实验能否检测出有效差异。

优质的测试任务应是用户日常会执行的真实工作，涉及受代码规范约束的代码，且需要添加需集成的内容：比如测试用例、接口、数据迁移或命令。

劣质的测试任务包括拼写错误修正、重命名或任何单行修复。这类任务完全可推导，两组测试结果会完全一致，用户会错误地认为整个AI指令层毫无价值。如果用户提出这类任务，需明确告知其弊端，并请求选择涉及代码规范的任务。

将最终确定的任务原文写入文件，每次运行都会完全复用该文件内容。

Step 3. Show the plan and get approval

步骤3：展示计划并获取用户批准

Report before spending anything: how many runs, which model, roughly what it will cost, and that the working tree will not be touched.

bash

python <skill>/scripts/run_ablation.py <repo> --task-file <task.md> --dry-run

The dry run also surfaces two things worth pausing on:

A dirty working tree. Worktrees are built from HEAD, so uncommitted edits are not under test. Offer to commit or stash first.
A build-dependency warning. Some repos import their own AI layer as source. A CLI that reads its skill markdown at build time breaks the moment those files go missing, and the user will read a compile error as an agent regression. Keep
```
--scope always
```
if this warns.

在开始执行前向用户汇报：运行次数、使用的模型、大致成本，以及不会改动工作树的说明。

bash

python <skill>/scripts/run_ablation.py <repo> --task-file <task.md> --dry-run

试运行还会发现两个需要注意的问题：

工作树未提交。测试基于HEAD创建worktree，因此未提交的修改不会被纳入测试范围。需主动提出帮用户提交或暂存修改。
构建依赖警告。部分仓库会将自身AI指令层作为源码导入。如果CLI在构建时读取skill markdown文件，那么当这些文件被移除时会触发编译错误，用户会将其误认为是Agent性能退化。如果出现此类警告，需保持
```
--scope always
```
参数不变。

Step 4. Run it

步骤4：执行实验

bash

python <skill>/scripts/run_ablation.py <repo> --task-file <task.md> --runs 2

This is the whole experiment. It builds a fresh worktree per run, strips the layer in the stripped arm, runs the same prompt in each, captures every diff, cleans up every worktree, and writes results to

.ablation/<timestamp>/

(gitignored).

Useful flags:

--runs 3

when the user intends to act on the result,

--scope all

to test the harder claim that skills and subagents have expired too,

--model

--jobs

for concurrency,

--runner

for a non-Claude agent that reads a prompt on stdin.

If an arm produced nothing usable, stop. An empty arm is a broken experiment, not a finding. Re-run before drawing anything from it.

bash

python <skill>/scripts/run_ablation.py <repo> --task-file <task.md> --runs 2

这就是完整的实验流程。它会为每次运行创建全新的worktree，在移除指令层的测试组中剥离AI指令，在两组中运行相同的提示词，捕获所有差异，清理所有worktree，并将结果写入

.ablation/<timestamp>/

实用参数：当用户打算根据结果采取行动时，使用

--runs 3

；使用

--scope all

可测试Skill和子Agent是否已失效；此外还有

--model

（指定模型）、

--jobs

（并发运行）、

--runner

（指定非Claude的Agent，从标准输入读取提示词）。

如果某一组未生成可用结果，需停止流程。空结果代表实验失败，而非有效结论。需重新运行实验后再进行分析。

Step 5. Grade

步骤5：评分

Read

references/comparison.md

before analysing. It is the rubric, and it contains the two things that make the difference between a real result and a confident wrong one: grade per rule rather than diffing the arms against each other, and grade blind to which arm a diff came from.

The short version:

Turn the always-loaded files into a numbered checklist of testable claims. Mark anything unfalsifiable ("write clean code") as exactly that.
Judge every run's diff against every claim:
```
followed
```
,
```
violated
```
, or
```
n/a
```
.
Only then join verdicts back to arms and read the pattern.

在分析前请阅读

references/comparison.md

文件。这是评分准则，其中包含两个关键要点，可区分真实结果与错误结论：按每条规则评分，而非直接对比两组结果；盲评，即不知道差异来自哪一组测试。

简化版评分流程：

将始终加载的指令文件转换为带编号的可测试声明清单。将无法证伪的内容（如“编写简洁代码”）标记为“无法证伪”。
对照每条声明判断每次运行的差异：“遵循”“违反”或“不适用”。
完成上述步骤后，再将评分结果与对应测试组关联，分析规律。

Step 6. Report

步骤6：汇报结果

Give the user a table, one row per rule, sorted so the actionable rows are first:

Pattern across runs	Verdict	Action
control follows, stripped violates	load-bearing	keep, rewrite shorter
both arms follow	model does this anyway	delete
both arms violate	ignored even when loaded	make it a hook or test, or delete
never applicable	untested	keep, no evidence either way
inconsistent within an arm	noise	more runs or a better task

Keep "untested" visually separate from "no difference". They look identical in the data and mean opposite things, and merging them is how a rule that protects a case this task never touched gets deleted.

为用户提供表格，每行对应一条规则，按可执行优先级排序：

跨运行规律	结论	操作建议
保留指令组遵循规则，移除指令组违反规则	核心必要规则	保留，可改写为更简洁的表述
两组均遵循规则	模型已原生支持该规则	删除
两组均违反规则	即使加载也被忽略	改为钩子或测试用例，或直接删除
从未适用	未测试	保留，暂无有效证据证明其价值或无用
同一组内结果不一致	随机误差	增加运行次数或更换更合适的测试任务

需将“未测试”与“无差异”在视觉上区分开。它们在数据中看起来相同，但含义完全相反——如果合并两者，可能会导致保护未被本次任务覆盖场景的规则被错误删除。

Step 7. Apply, with the user's approval

步骤7：在用户批准后执行修改

Never edit the rules file unattended. Propose the edit, show the diff, wait.

Re-add or keep one line at a time, only for rules with observed evidence, and prefer a test, then a hook, then an on-demand instruction, and only then an always-loaded line. Finish by re-running

map_layer.py

so the new always-loaded total sits next to the old one.

绝不能擅自编辑规则文件。需提出修改建议，展示差异，等待用户确认。

仅针对有观测证据支持的规则，逐行重新添加或保留；优先选择测试用例，其次是钩子，然后是按需加载指令，最后才是始终加载的指令行。完成修改后，重新运行

map_layer.py

，让用户看到新的始终加载指令集资源成本与旧成本的对比。

Honest framing to give the user

需向用户说明的客观情况

One probe task is a data point, not a verdict. Encourage a second task on a different part of the codebase before deleting anything large.
A null result is a real result. If the arms match, that part of the layer has genuinely expired and can go.
The reverse is also true. Do not let one clean run justify deleting rules for cases this task never exercised: security, compliance, release procedure.
Existing code substitutes for the rules file. A stripped run copies conventions from neighbouring code when there is a neighbour to copy. The same rule can hold in an edited file and break in a new one. Weight new-file evidence more heavily and say which kind each verdict rests on.
Cheaper and smaller models lean on instructions more than frontier models do. A layer that looks redundant under a frontier model may still be carrying a cheaper one. If the team runs a mix, ablate against the weakest model in use.

单个测试任务仅为一个数据点，而非最终结论。建议在代码库的不同部分执行第二个测试任务后，再删除大量规则。
无差异结果也是有效结论。如果两组测试结果一致，说明该部分AI指令层确实已失效，可删除。
反之亦然。不要仅凭一次顺利运行就删除针对未被本次任务覆盖场景的规则：比如安全、合规、发布流程相关规则。
现有代码可替代规则文件。移除指令层的测试会从相邻代码中复制规范（如果有可复制的相邻代码）。同一条规则在已有文件中可能有效，但在新文件中可能失效。需更重视新文件中的证据，并说明每条结论基于哪种场景。
低成本小模型比前沿模型更依赖指令。在前沿模型下看似冗余的指令层，可能仍在支撑低成本模型。如果团队混合使用多种模型，需针对正在使用的最弱模型进行消融测试。

Resources

资源

```
scripts/map_layer.py
```
: read-only inventory of the layer, agent-agnostic.
```
scripts/run_ablation.py
```
: runs both arms and collects the diffs.
```
--help
```
lists every flag. Never read either script into context; only their output.
```
references/comparison.md
```
: the grading rubric. Read it before Step 5.

```
scripts/map_layer.py
```
：只读的AI指令层清单工具，兼容多种Agent。
```
scripts/run_ablation.py
```
：执行两组测试并收集差异的脚本。
```
--help
```
可查看所有参数。无需将脚本内容纳入上下文，仅需关注其输出。
```
references/comparison.md
```
：评分准则。请在步骤5前阅读。