term-check

Compare original and translation side by side

🇺🇸

Original

English

🇨🇳

Translation

Chinese

A name earns its place when a reader who has never seen the docs guesses right. This probe measures that: one scenario, three candidate terms swapped in, three models, two runs each. Three arms are the default — when only two candidates are on the table, propose a third synonym and run all three.

当从未看过文档的读者能猜对某个名称的含义时，该名称才算是合格的。本测试方法正是衡量这一点：一个场景，三个候选术语轮流替换，三个模型，每个模型运行两次。默认测试三个候选术语——如果只有两个候选，建议补充第三个同义词后再同时测试三个。

Steps

步骤

Write the scenario to a file: the domain in two sentences, the shape around the word, and
```
{TERM}
```
where the candidate goes. State only what a stranger needs to place the word, because a scenario that names the meaning under test answers its own question. Close with a fixed answer shape: one sentence plus
```
confidence: high|medium|low
```
, so the runs stay comparable.

Run the probe:

bash .agents/skills/term-check/probe.sh <scenario-file> <term-a> <term-b> <term-c>

Six runs per term, in parallel, about one minute.

Score every run against what the word must convey: right, partly right, or wrong. Keep each model's own confidence line beside its score.
Report the scenario, then the table. Quote the scenario exactly as the models got it, in a fenced block above the table, so the reader judges the question before the answers.
One row per model and run, one column per term. Every cell reads
```
<icon> (<confidence>) -> <guess>
```
: the score as ✅ right, ⚠️ partly right or ❌ wrong, the model's own confidence word, and its guess in a few words, so the reader sees what the model understood. Right answers decide first, confidence second.
Model
term-a
term-b
term-c
Opus, run 1 ✅ (medium) -> moves it to the next day's plan ❌ (low) -> deletes it at the seal ⚠️ (low) -> keeps it somewhere unnamed
Offer another round when the winner is weak. A winner is weak when it takes no
```
high
```
confidence, or when it wins only because the other arms collapsed. Say so, name 3 or 4 fresh synonyms, ask the user whether to run them, and sharpen the scenario in the same breath, because a thin scenario holds every confidence down.

Model	`term-a`	`term-b`	`term-c`
Opus, run 1	✅ (medium) -> moves it to the next day's plan	❌ (low) -> deletes it at the seal	⚠️ (low) -> keeps it somewhere unnamed

Done when all six runs per term carry an icon, a confidence and a guess, the scenario stands above the table, and the table names a winner, a tie, or a weak winner with a fresh set of synonyms offered.

编写场景文件：用两句话说明领域背景，描述术语的使用语境，并用
```
{TERM}
```
标记候选术语的位置。只写出陌生读者理解术语所需的信息，因为如果场景直接点明了要测试的含义，就相当于自问自答。结尾固定格式为一句话加上
```
confidence: high|medium|low
```
，确保不同测试的结果具有可比性。

运行测试：

bash .agents/skills/term-check/probe.sh <scenario-file> <term-a> <term-b> <term-c>

每个术语运行6次，并行执行，耗时约1分钟。

为每次运行评分：根据术语需要传达的含义，判定结果为正确、部分正确或错误。将模型自身给出的置信度与评分对应记录。
展示场景与结果表格：将模型收到的场景原文用围栏块引用在表格上方，让读者先判断问题是否合理，再查看答案。
表格中每行对应一个模型的一次运行，每列对应一个术语。每个单元格格式为
```
<图标> (<confidence>) -> <猜测内容>
```
：其中评分用✅表示正确，⚠️表示部分正确，❌表示错误，同时包含模型给出的置信度词汇，以及模型的简短猜测内容，便于读者了解模型的理解情况。优先依据正确答案判断，其次参考置信度。
模型
term-a
term-b
term-c
Opus, run 1 ✅ (medium) -> moves it to the next day's plan ❌ (low) -> deletes it at the seal ⚠️ (low) -> keeps it somewhere unnamed
若胜出术语说服力不足，建议再次测试：当胜出术语没有获得
```
high
```
置信度，或仅因其他候选表现极差而胜出时，说明其说服力不足。此时应告知用户这一点，提出3-4个新的同义词，询问用户是否要再次测试，同时优化场景描述——因为模糊的场景会拉低所有结果的置信度。

模型	`term-a`	`term-b`	`term-c`
Opus, run 1	✅ (medium) -> moves it to the next day's plan	❌ (low) -> deletes it at the seal	⚠️ (low) -> keeps it somewhere unnamed

当每个术语的6次运行都有对应的图标、置信度和猜测内容，场景展示在表格上方，且表格明确指出胜出术语、平局结果，或说服力不足的胜出术语并提供新的同义词选项时，测试完成。

Reading the result

结果解读

Outcome	What it means
All arms score alike	First suspect the scenario, not the words: outcomes this similar usually mean the scenario hints too strongly and answers its own question. Re-read it, name the leaking phrase, and ask the user whether to run another probe with the scenario changed. Only when the scenario is clean does a tie mean the word does no work — then pick on other grounds: brevity, parallel with its siblings, the glossary
One arm wins	That word carries the meaning. Take it
All arms score badly	The words are fine and the shape is wrong. Fix the shape, then probe again

结果	含义
所有候选术语得分相近	首先怀疑场景是否存在问题，而非术语本身：这种结果通常意味着场景暗示性过强，相当于自问自答。重新检查场景，找出泄露答案的表述，询问用户是否要修改场景后再次测试。只有当场景无误时，平局才说明术语无法有效传达含义——此时可根据其他标准选择：简洁性、与同类术语的一致性、词汇表规范
某一个候选术语胜出	该术语能有效传达含义，可选用
所有候选术语得分都很低	术语本身没问题，是场景语境设计有误。修正语境后重新测试

Limits

局限性

Each run starts from an empty conversation in a neutral directory, so no repo doc reaches the model. The global

~/.claude/CLAUDE.md

still loads. It holds no domain vocabulary and it loads for every arm alike, so a comparison between arms holds while an absolute score does not.

A model guesses from the whole scenario, so a scenario that leaks the answer scores every candidate right. Arms that tie are the signal to re-read the scenario before trusting the tie.

The confidence word is the model's own report, written because the scenario asks for it. No number stands behind it and nothing calibrates it, so read it as a tone and let the score carry the verdict. The measured number is the agreement rate: how many of the runs land on the same meaning. Raise the run count in

probe.sh

to sharpen it.

每次运行都从空对话开始，在中性目录中执行，因此不会加载仓库文档。但全局的

~/.claude/CLAUDE.md

仍会加载。该文件不包含领域词汇，且对所有候选术语的测试都加载相同内容，因此候选术语之间的对比是有效的，但绝对评分不具备参考性。

模型会根据整个场景进行猜测，若场景泄露答案，所有候选术语都会得到高分。当候选术语得分相同时，需先重新检查场景，再信任平局结果。

置信度词汇是模型自行给出的，因为场景要求它提供该信息。置信度没有数值支撑，也未经过校准，因此仅作为参考语气，最终结论以评分为准。可衡量的指标是一致率：即有多少次运行得出了相同的含义。可修改

probe.sh

中的运行次数来提高结果的准确性。