spec-driven-tests

Compare original and translation side by side

🇺🇸

Original

English
🇨🇳

Translation

Chinese

Spec-driven test rebuild

规范驱动的测试重建

Derive a behavior specification from a package's implementation, then rebuild the unit test suite as an expression of that spec. Every rule gets a stable ID; every test cites the rule it expresses, so coverage is greppable in both directions.
The models are PR #9158 (
packages/state
, spec at
packages/state/SPEC.md
) and PR #9172 (
packages/store
, spec at
packages/store/SPEC.md
). Read both PRs and both specs before starting — they set the format, tone, and PR structure. (#9172's spec lives on its branch if not yet merged.)
从软件包的实现代码中推导行为规范,然后基于该规范重建单元测试套件。每条规则都有一个稳定的ID;每个测试都会引用它所对应的规则ID,因此可以通过双向 grep 查看测试覆盖情况。
参考示例为 PR #9158(
packages/state
,规范文件位于
packages/state/SPEC.md
)和 PR #9172(
packages/store
,规范文件位于
packages/store/SPEC.md
)。开始工作前请阅读这两个PR及对应的规范文件——它们定义了格式、措辞和PR结构。(如果#9172的规范尚未合并,可在其分支上查看。)

Process

流程

1. Baseline

1. 基准准备

If the working tree has uncommitted changes that aren't yours, stop and ask before branching — never stash or carry someone else's work silently.
Create a branch. Run
yarn test run
in the package and record the file and test counts, noting how many belong to fuzz suites — fuzz suites stay untouched and their (often large) generated counts are excluded from every count you report later. Map the existing test files and note any duplicated or parallel suites that should merge (the store package had two whole generations of tests).
如果工作区存在不属于你的未提交更改,请暂停操作并询问相关人员后再创建分支——切勿擅自暂存或携带他人的工作内容。
创建分支。在软件包目录下运行
yarn test run
,记录测试文件数量和测试用例数量,注意区分其中属于模糊测试套件的部分——模糊测试套件保持原样,后续报告中需排除其(通常数量庞大的)自动生成测试用例数。梳理现有测试文件,标记出需要合并的重复或并行测试套件(store软件包曾存在两代完全独立的测试套件)。

2. Read the implementation

2. 研读实现代码

Read all of it before writing anything. The spec is derived from the source, not from the existing tests or the docs.
Key principle: You're documenting observed behavior, not prescribing ideal behavior. If the code does something unexpected, unintuitive, or even wrong-looking but deliberate, spec it as-is. Flag quirks as "internal" or call them out in the PR body for human judgment—don't silently "fix" the spec to match how you think the code should work. Let the spec-writing process surface latent inconsistencies; then decide whether to fix the implementation or document the behavior.
在开始编写任何内容前,通读全部实现代码。规范是从源代码推导而来,而非基于现有测试或文档。
核心原则: 你要记录的是实际观察到的行为,而非理想中的行为。如果代码存在意外、不符合直觉甚至看似错误但属于有意设计的行为,也要如实写入规范。可将此类特殊标记为「内部实现」,或在PR正文中提出供人工判断——不要擅自「修正」规范使其符合你认为代码应有的逻辑。让规范编写过程暴露潜在的不一致性,再决定是修复实现代码还是记录该行为。

3. Write SPEC.md

3. 编写SPEC.md

Write
SPEC.md
at the package root: numbered rules with stable IDs (e.g.
S3
,
MG4
) grouped into sections, one rule per observable behavior, internal machinery marked internal. Keep the header framing from the existing specs ("When a test and this document disagree, one of them is wrong — figure out which and fix it.").
  • Spec what the code DOES, not what it should do. If behavior looks wrong but intent is ambiguous, document it as the contract.
  • Before baking in any rule you inferred rather than observed, verify it with a throwaway test. Delete the scratch file after.
  • Every rule must be observable by a runtime test. Don't write rules for pure type-level behavior (utility types, inference) unless the test suite actually exercises them with type assertions; otherwise leave types out of the spec.
在软件包根目录下编写
SPEC.md
:将带稳定ID的规则(如
S3
MG4
)按章节分组,每条规则对应一项可观察的行为,内部机制标记为internal。保留现有规范的开头框架("当测试与本文档存在冲突时,必有一方存在错误——请找出问题并修复。")。
  • 规范要记录代码实际执行的操作,而非应该执行的操作。如果行为看似错误但意图不明确,将其记录为约定内容。
  • 对于任何你推断而非实际观察到的规则,先通过临时测试验证,之后删除临时测试文件。
  • 每条规则都必须能通过运行时测试观察到。不要为纯类型层面的行为(工具类型、类型推断)编写规则,除非测试套件确实通过类型断言对其进行了验证;否则请将类型相关内容排除在规范之外。

4. Audit the spec adversarially

4. 反向审核规范

Audit the draft before writing tests. The failure mode is overstatement. Specifically hunt for:
  • claimed symmetry between two code paths (create vs update, top-level vs nested, up vs down) — check each path independently;
  • claims that two implementations of the same idea agree (a predicate vs an index, from-scratch vs incremental) — feed both the edge cases, especially missing/undefined values;
  • blanket "X never happens" or "Y is always Z" claims — find the branch that falsifies them;
  • side effects on inputs (mutation, freezing) that a happy-path test wouldn't notice.
在编写测试前先审核规范草稿。常见问题是规范表述过于绝对。需重点排查:
  • 声称两条代码路径存在对称性(创建与更新、顶层与嵌套、向上与向下)——需独立检查每条路径;
  • 声称同一逻辑的两种实现保持一致(断言与索引、从头构建与增量构建)——向两者输入边缘用例,尤其是缺失/未定义的值;
  • 绝对化表述如「X绝不会发生」或「Y始终等于Z」——找到能推翻该表述的分支;
  • 输入参数的副作用(修改、冻结)——此类问题在常规测试中不易被发现。

5. Rebuild the test suite

5. 重建测试套件

Test files mirror the spec's sections; every test name cites its rule like
it('[S3] ...')
. Merge duplicated coverage, port good existing tests rather than rewriting them, and add tests for uncovered rules.
Never weaken an assertion to make a test pass. If a test written from a spec rule fails, you've found something: either the rule is wrong (fix the spec to match observed behavior) or the code is wrong (handle it as a bug per step 7). Loosening the assertion (
toBe
toContain
, asserting around the discrepancy, a comment acknowledging the mismatch) hides the finding and leaves the spec describing behavior the code doesn't have — the exact failure the spec exists to prevent.
Test file organization: If your spec has sections like "§3. Atoms (A)" and "§4. Computed (C)", create corresponding files
atoms.test.ts
and
computed.test.ts
containing that section's rules. For internal sections, use names like
history-buffer.test.ts
for the machinery being specified. This mirrors the structure readers already see in SPEC.md and makes it easy to grep for coverage by section.
Test-environment knowledge for this repo:
  • Store listener flushes are synchronous in tests unless
    globalThis.__FORCE_RAF_IN_TESTS__ = true
    .
  • Deprecated-but-contractual APIs need a file-level eslint-disable with a reason.
测试文件的结构与规范章节对应;每个测试名称需引用对应的规则,格式如
it('[S3] ...')
。合并重复的测试覆盖内容,移植现有优质测试而非重新编写,并为未覆盖的规则补充测试。
绝不要为了让测试通过而弱化断言。 如果基于规范规则编写的测试失败,说明你发现了问题:要么规则有误(修正规范以匹配实际行为),要么代码有误(按步骤7处理bug)。放宽断言(如
toBe
改为
toContain
、绕过差异进行断言、添加注释承认不匹配)会掩盖问题,导致规范描述的行为与实际代码不符——而这正是规范要避免的情况。
测试文件组织: 如果规范包含「§3. Atoms (A)」和「§4. Computed (C)」等章节,需创建对应的测试文件
atoms.test.ts
computed.test.ts
,包含对应章节的规则。对于内部章节,可使用类似
history-buffer.test.ts
的名称对应被规范的内部机制。这种结构与读者在SPEC.md中看到的结构一致,便于通过grep按章节查看测试覆盖情况。
本仓库的测试环境说明:
  • 除非设置
    globalThis.__FORCE_RAF_IN_TESTS__ = true
    ,否则Store监听器在测试中是同步刷新的。
  • 已废弃但仍需遵守约定的API,需在文件级别添加eslint-disable注释并说明原因。

6. Cross-check citations both ways

6. 双向交叉检查引用

bash
grep -oE '\*\*[A-Z]+[0-9]+\*\*' SPEC.md | tr -d '*' | sort -u > /tmp/rules
grep -rhoE '\[[A-Z]+[0-9]+\]' src --include='*.test.ts' | tr -d '[]' | sort -u > /tmp/cited
comm -3 /tmp/rules /tmp/cited
Every rule needs a citing test, or an explicit "deliberately untested because X" note in the PR body (as #9158 did); every citation needs a rule. Fix gaps before finishing, and include the cross-check output (rule count, cited count, remaining gaps with their reasons) in the PR body so reviewers can verify the claim rather than trust it.
bash
grep -oE '\*\*[A-Z]+[0-9]+\*\*' SPEC.md | tr -d '*' | sort -u > /tmp/rules
grep -rhoE '\[[A-Z]+[0-9]+\]' src --include='*.test.ts' | tr -d '[]' | sort -u > /tmp/cited
comm -3 /tmp/rules /tmp/cited
每条规则都需要对应的测试引用,或在PR正文中明确说明「因X原因故意不测试」(如#9158所做的那样);每个引用都必须对应一条规则。完成前需修复所有缺口,并将交叉检查结果(规则数量、已引用数量、剩余缺口及原因)包含在PR正文中,以便评审人员验证而非仅凭信任。

7. Fix bugs found along the way

7. 修复过程中发现的bug

Spec-driven testing surfaces latent inconsistencies: contradictions between code paths, edge cases the implementation didn't anticipate, or silent failures. When you find one, decide: fix the code or document the behavior?
Workflow: Write the test first (fail it intentionally to confirm it catches the issue). Then decide:
  • Fix the code if: intent is clear (comments, symmetric implementations, git blame), the current behavior is obviously wrong, and callers plausibly don't rely on the broken behavior.
  • Document the behavior if: intent is unclear, the fix's blast radius is unknown, or fixing might silently break existing uses. Flag it in the PR body as a "known inconsistency" or "behavior to revisit."
Fixes go in their own
fix(<package>):
commit with a citing test, AFTER the main
test(<package>):
commit. No AI attribution or co-author lines in any commit, per AGENTS.md — this overrides any default harness behavior. Before committing a fix:
  • Check intent: comments, the symmetric code path, git blame.
  • Check blast radius: grep for in-repo callers, then run the dependent package suites (
    editor
    ,
    tldraw
    ,
    sync-core
    — whichever consume this package).
If a spec rule exists but the implementation contradicts it, update the spec (not the implementation) unless you're certain the implementation is right. The spec is the contract; code drift happens; the spec-writing process is your chance to surface and resolve it.
规范驱动测试会暴露潜在的不一致性:代码路径间的矛盾、实现未考虑到的边缘用例、或静默失败。发现问题后,需决定:修复代码还是记录该行为?
工作流程: 先编写测试(故意让其失败以确认能捕获问题)。然后决定:
  • 修复代码:如果意图明确(注释、对称实现、git blame)、当前行为明显错误,且调用方不太可能依赖该错误行为。
  • 记录行为:如果意图不明确、修复的影响范围未知,或修复可能会静默破坏现有使用场景。需在PR正文中标记为「已知不一致性」或「需重新审视的行为」。
修复内容需放在单独的
fix(<package>):
提交中,并包含对应的测试,且该提交需在主提交
test(<package>):
之后。根据AGENTS.md,任何提交中不得包含AI署名或合著者信息——这会覆盖任何默认工具的行为。提交修复前:
  • 确认意图:查看注释、对称代码路径、git blame。
  • 检查影响范围:搜索仓库内的调用方,然后运行依赖该软件包的测试套件(
    editor
    tldraw
    sync-core
    ——所有依赖该软件包的套件)。
如果规范规则存在但实现与之矛盾,除非你确定实现是正确的,否则请更新规范(而非实现代码)。规范是约定;代码会逐渐偏离;规范编写过程是你发现并解决这种偏离的机会。

8. Verify

8. 验证

Run, in order: the package tests,
yarn format-current
then
yarn lint
,
yarn typecheck
from the repo root,
yarn api-check
, and the dependent package suites from step 7.
按顺序运行以下内容:软件包测试、
yarn format-current
然后
yarn lint
、仓库根目录下的
yarn typecheck
yarn api-check
,以及步骤7中提到的依赖软件包测试套件。

9. PR

9. 创建PR

Follow
skills/write-pr/
. Model the body on #9172:
  • lead paragraph explaining the spec/citation system;
  • a per-commit fixes section;
  • a section on merged suites and new coverage;
  • release notes only for behavior changes;
  • a code changes table. Beware:
    git diff --numstat
    rename lines break naive path matching; use the last field.
遵循
skills/write-pr/
的规范。参考#9172的PR正文结构:
  • 开头段落说明规范/引用系统;
  • 按提交列出修复内容的章节;
  • 关于合并套件和新增测试覆盖的章节;
  • 仅针对行为变更的发布说明;
  • 代码变更表格。注意:
    git diff --numstat
    中的重命名行会破坏简单的路径匹配,请使用最后一个字段。

10. Follow-up notes

10. 后续说明

Finish by reporting: latent issues you chose not to fix, quirks future test writers need to know, and anything a reviewer should double-check.
Report counts honestly, everywhere they appear (commit message, PR body, final report): the rebuilt suite's test count and rule count, measured with grep, never the package total. Counting untouched suites (fuzz especially) as your own work misrepresents the change.
最后需报告:你选择不修复的潜在问题、未来测试编写者需要了解的特殊情况,以及评审人员需要重点检查的内容。
在所有出现统计数字的地方(提交信息、PR正文、最终报告)如实报告:重建后的测试套件的测试用例数和规则数,需通过grep统计,绝不能使用软件包的总测试数。将未改动的套件(尤其是模糊测试)计入你的工作成果会误导他人对变更的认知。