test-value-evidence
Compare original and translation side by side
🇺🇸
Original
English🇨🇳
Translation
ChineseTest-value evidence
测试价值证据
A passing test proves little unless it detects the relevant wrong behavior.
Assess the changed contracts and their directly related tests, not the entire
repository. Discover and follow the repository's instructions, pinned toolchain,
test commands, assertion conventions, and execution permissions. This skill
does not authorize edits in a read-only review, execution of untrusted code,
workflow dispatch, publication, or access to another session's checkout.
一个通过的测试若不能检测相关的错误行为,其证明力就很有限。
评估已变更的契约及其直接相关的测试,而不是整个
仓库。查找并遵循仓库的说明、固定工具链、
测试命令、断言约定和执行权限。本 skill
不授权在只读审查中进行编辑、执行不可信代码、
调度 workflow、发布内容,或访问另一个会话的 checkout。
Inputs and boundaries
输入与边界
Discover the requested diff/revision, production entry points, tests, fixtures,
and available execution artifacts. If the change cannot be identified, report
the missing scope instead of grading the workspace. Existing unchanged tests
can suffice; a source-only diff is not automatically untested.
Use for general existing-suite blind spots,
for curated per-test readiness/quality, and existing repository mutation tooling
for mutation campaigns. Do not start a second mutation engine or require
full-suite mutation testing for every change. This skill is self-contained:
sibling skills are optional routing destinations, not required dependencies.
test-gap-analysisgrade-tests发现所请求的 diff/修订版本、生产入口点、测试、fixtures
以及可用的执行产物。如果无法确定变更,应报告
缺失的范围,而不是对工作区进行评级。未发生变更的现有测试
可能已经足够;仅包含源代码的 diff 并不自动意味着未经过测试。
对现有测试套件的一般盲点使用 ,对精选的逐项测试准备度/质量使用
,对变异测试活动使用仓库现有的变异工具。不要启动第二个变异引擎,也不要要求
每次变更都进行完整测试套件的变异测试。此 skill 是自包含的:
同级 skill 仅是可选的路由目标,不是必需依赖。
test-gap-analysisgrade-testsWorkflow
工作流
-
Bind the contract to the tested revision. For a PR, record its merge base and exact head SHA. For a commit, record the selected parent and commit; identify the parent explicitly for a merge commit. For uncommitted work, record the base SHA plus a reproducible patch identity (saved patch/hash, including relevant untracked files), not a fictitious committed revision. Evidence expires after relevant source, tests, helpers, configuration, or package changes unless revalidated.For each distinct behavior change, map:. Derive the expected outcome independently from requirements or supported API/protocol semantics, not the implementation or the test's current value. Check build/package/configuration behavior, deleted/renamed code, removed tests, and shipping consumers as well as source. Include positive, negative, boundary, compatibility, cancellation, cleanup, and multi-module cases only where relevant to the changed contract.
requirement -> input/sequence -> expected public outcome -> production entry point -> named test/data case -> distinguishing assertionA mock of the changed component, self-comparison, or expected value computed by the same production algorithm does not provide an independent oracle. -
Choose the meaningful counterfactual.
Change Counterfactual Required observation Bug fix Keep the regression test and restore the actual pre-fix defect. The test fails for that defect, then passes with the fix. New feature Disable/remove or corrupt the behavior, retaining the API shape and setup needed to build. An assertion detects absent/wrong behavior, then passes with the feature. Stronger/test-only coverage Introduce one plausible wrong public outcome. The strengthened test detects it; the unmodified product passes. Behavior-preserving refactor Compare the preserved contract before and after; optionally probe a relevant fault. Preservation tests pass on both revisions; passing before is expected. Documentation/generated metadata only Establish whether an executable contract changed. Not applicable only with a concrete reason; generated or build metadata can change behavior. For a fix, prefer the real pre-fix implementation with the same regression test, setup, dependencies, and inputs. If new APIs prevent that combination from compiling, restore only the defective behavior in the head version. Call this a targeted counterfactual, not a run on the base revision. Record its exact patch/hash. Never change assertions to manufacture red.For a feature, a missing-symbol compiler error is not behavioral proof. Choose a buildable no-op, disabled registration, old branch, or wrong output. A compiler/analyzer diagnostic can be valid behavior only when an executed harness asserts that exact diagnostic or its absence.Prefer small plausible faults: restore the old boundary, drop a required output, skip a callback, ignore an option, or return the wrong value. An equivalent mutation is not a gap. A crash, timeout, or exception counts only when it is the intended contract signal and diagnostics attribute it to the fault, not setup, discovery, flakes, or infrastructure. -
Execute paired runs only when authorized. Implementation normally requires execution; read-only/advisory review normally requires inspection. A review request alone does not permit source edits or builds/tests. Never run untrusted code with privileged credentials.In an authorized implementation/verification environment:
- Use an owned isolated checkout or experiment directory. Preserve the user's dirty files and other sessions/main checkout. Preflight concurrent processes and shared outputs/resources; do not kill unrelated processes.
- Run the smallest relevant selection on the correct implementation. Confirm the named tests and data cases actually execute and pass.
- Apply only the chosen defective behavior in the isolated experiment. Rebuild and rerun the same tests, assertions, data, dependencies, and applicable runtime configuration. Capture the named failure, assertion, expected/actual observation, process exit, and diagnostics.
- Restore the correct implementation, rebuild, and rerun to confirm green again. Verify protected source/test bytes, remove only owned temporary resources, and never deliver the counterfactual patch in the product.
Paired runs must not reuse stale build/test outputs or the other variant's effective caches. Use separate outputs/caches or clean only owned outputs and rebuild; do not rely onor a runner's success exit alone. Record selected/executed/passed/failed/skipped counts for each run. If a runner does not expose a count, say unknown rather than inventing it. Zero selection/execution, skips, missing tools, authentication, setup/build, unrelated failures, and unattributed timing flakes leave proof unverified.--no-buildFor cross-component/acceptance behavior, identify the producer-consumer contract first: ownership, lifecycle, handshake/no-handshake, exit semantics, fallback, compatibility, cancellation, and multi-module aggregation where applicable. Record the exact shipping binaries/packages/extensions and release branches. Rebuild/repack each variant, isolate effective package caches, prove which variant the real shipping consumer loaded, and run a representative composed command. Unit mocks or source-compatible compilation alone do not replace shipping-layout validation. Exercise applicable normal, suppressed, partial, legacy/incompatible, cancellation, nonzero-exit, zero-result, fallback, and multi-module paths. Keep rollout/servicing ownership explicit; a feature flag does not lower the evidence bar.In read-only/advisory mode, trace both revisions, setup/helpers, inputs, and assertions without edits or execution unless separately permitted. Label conclusions static predictions, never executed proof. Existing artifacts count only when their revisions/patches, loaded variant, selection, counts, and failure attribution match. Green CI or an author's claim alone is not red-green evidence. Say what is inaccessible or missing. -
Report only the evidence obtained. Put a compact Test-value evidence section in the handoff or authorized review/description. State base/parent and head SHAs or tested patch identities. Use one row per changed contract, group tests protecting the same contract, and identify unassessed groups.
Contract / test case Counterfactual Without behavior Correct implementation Evidence / gap Boundary defect / named case Restore old comparison; patch identity Executed: assertion failed Executed: passed Commands, exits, expected/actual, counts, artifacts New output / named case Skip output write Static: predicts failure Not executed Source/assertion locations; paired execution missing Use executed: detected, executed: survived, static: predicts detection, static: predicts survival, unverified, equivalent, or not applicable accurately. Missing red/green or incomplete selection leaves paired proof unverified. Report executed: survived only for an executed, non-equivalent fault; do not count equivalent edits as gaps. Preservation passes for a refactor are valid evidence, not missing-red defects.Include exact commands/exits, toolchain/framework/configuration, revision and counterfactual patch identities, per-run counts, and relevant artifact paths. Distinguish execution performed now from verified existing artifacts. Never combine static predictions with measured mutation counts/scores. Evidence for a final PR head does not prove every intermediate commit; assess a requested commit against its parent separately.Missing evidence is not automatically a proven defect or an inline review thread. Publication follows the caller's existing authorization gate; do not create a new GitHub object just to store a report or claim instructions enforce hooks, CI, or publication.
-
将契约绑定到被测试的修订版本。 对于 PR,记录其 merge base 和准确的 head SHA。对于 commit,记录所选 parent 和 commit; 对于 merge commit,要明确标识 parent。对于未提交的工作, 记录 base SHA 以及可复现的 patch 标识(保存的 patch/哈希, 包括相关的未跟踪文件),而不是虚构一个已提交的修订版本。 除非重新验证,否则相关源代码、测试、辅助代码、配置或 包发生变化后,证据即失效。对于每项不同的行为变更,映射:。 从需求或受支持的 API/协议语义中独立推导预期结果,而不是根据实现或测试的当前值。 除源代码外,还要检查构建/打包/配置行为、删除或重命名的代码、已移除的 测试以及实际交付的消费者。仅在与变更契约相关时, 纳入正向、负向、边界、兼容性、取消、清理和多模块场景。
requirement -> input/sequence -> expected public outcome -> production entry point -> named test/data case -> distinguishing assertion对已变更组件的 mock、自比较,或使用同一生产算法计算出的预期值, 均不能提供独立 oracle。 -
选择有意义的反事实。
变更 反事实 必需观察结果 缺陷修复 保留回归测试并恢复实际的修复前缺陷。 测试因该缺陷而失败,并在应用修复后通过。 新功能 在保留构建所需的 API 形态和设置的同时,禁用/移除或破坏该行为。 断言检测到行为缺失/错误,并在启用该功能后通过。 加强测试/仅测试覆盖 引入一个合理但错误的公开结果。 加强后的测试可检测该错误;未修改的产品通过。 保持行为不变的重构 比较重构前后所保留的契约;可选地探测一个相关故障。 契约保持测试在两个修订版本上均通过;在修复前通过是预期结果。 仅文档/生成的元数据 确定可执行契约是否发生变化。 只有在有具体理由时,才可标为不适用;生成内容或构建元数据也可能改变行为。 对于修复,应优先使用真实的修复前实现,并搭配相同的回归测试、 设置、依赖项和输入。如果新 API 导致该组合无法编译, 则只在 head 版本中恢复有缺陷的行为。 将其称为 定向反事实,而不是在 base 修订版本上运行。 记录其准确的 patch/哈希。绝不能通过更改断言来人为制造红灯。对于新功能,缺少符号导致的编译器错误不能作为行为证明。 应选择可构建的 no-op、禁用注册、旧分支或错误输出。 只有在已执行的 harness 明确断言该诊断或其不存在时, 编译器/分析器诊断才可作为有效行为证据。优先选择小而合理的故障:恢复旧边界、丢弃必需输出、跳过回调、 忽略某个选项,或返回错误的值。 等价变异不构成缺口。崩溃、超时或异常只有在属于预期的契约信号, 且诊断将其归因于该故障,而不是设置、测试发现、flake 或基础设施问题时,才计入有效证据。 -
仅在获得授权时执行配对运行。 实施任务通常需要执行; 只读/建议性审查通常只需检查。 单纯的审查请求并不允许编辑源代码或构建/测试。 绝不能在拥有特权凭据的情况下运行不可信代码。在获准的实施/验证环境中:
- 使用自有隔离的 checkout 或实验目录。保留 用户的脏文件,以及其他会话/主 checkout。预先检查并发 进程和共享输出/资源;不要终止无关进程。
- 在正确实现上运行最小相关测试集。 确认指定测试和数据用例确实执行并通过。
- 仅在隔离实验中应用所选的有缺陷行为。 重新构建并以相同的测试、断言、数据、依赖项和 适用运行时配置再次运行。捕获指定失败、断言、 预期/实际观察结果、进程退出状态和诊断信息。
- 恢复正确实现,重新构建并再次运行以确认恢复为绿灯。 验证受保护的源文件/测试字节,仅删除自有临时资源, 且绝不将反事实补丁交付到产品中。
配对运行不得复用陈旧的构建/测试输出或另一变体的 实际生效缓存。使用独立的输出/缓存,或仅清理自有输出 并重新构建;不要依赖,也不要仅凭 runner 成功退出就下结论。记录每次运行 selected/executed/passed/failed/skipped 计数。如果 runner 不提供计数,应写为 unknown,而不是编造。选择或执行数量为零、 跳过、缺少工具、认证问题、设置/构建问题、 无关失败和无法归因的时序 flake,都会使证明保持未验证。--no-build对于跨组件/验收行为,先确定生产者-消费者契约: 所有权、生命周期、握手/无握手、退出语义、 回退、兼容性、取消,以及适用时的多模块聚合。 记录实际交付的二进制文件/包/扩展和 发布分支。重新构建/重新打包每个变体,隔离实际生效的包缓存, 证明真实交付消费者加载的是哪个变体,并运行 有代表性的组合命令。仅使用单元 mock 或源代码兼容编译 不能替代交付布局验证。执行适用的正常、 被抑制、部分、旧版/不兼容、取消、非零退出、 零结果、回退和多模块路径。明确 rollout/servicing 的责任归属; feature flag 不会降低证据标准。在只读/建议性模式下,应追踪两个修订版本、设置/辅助项、输入和 断言,不进行编辑或执行(除非另行获准)。应将结论标为 静态预测,而不能称为已执行的证明。只有当现有产物的修订版本/patch、加载的变体、选择范围、计数和 失败归因全部匹配时,才能将其计为证据。CI 绿灯或作者的说法本身 不构成 red-green 证据。说明哪些内容无法访问或缺失。 -
仅报告已获得的证据。 在交接材料或获准的审查/描述中加入简明的 测试价值证据 章节。说明 base/parent 和 head SHA 或已测试 patch 的标识。每个变更契约使用一行, 将保护同一契约的测试归为一组,并标出未评估的测试组。
契约/测试用例 反事实 缺少行为时 正确实现 证据/缺口 边界缺陷/指定用例 恢复旧比较逻辑;patch 标识 已执行:断言失败 已执行:通过 命令、退出状态、预期/实际结果、计数、产物 新输出/指定用例 跳过输出写入 静态:预测失败 未执行 源代码/断言位置;缺少配对执行 准确使用 executed: detected、executed: survived、static: predicts detection、static: predicts survival、unverified、equivalent 或 not applicable。缺少红灯/绿灯或不完整的选择会使配对证明保持未验证。仅当故障已执行且非等价时,才报告 executed: survived;不要把等价编辑计为缺口。重构中的契约保持测试通过属于有效证据,而不是缺少红灯的缺陷。包括准确的命令/退出状态、工具链/框架/配置、修订版本 和反事实 patch 标识、每次运行的计数,以及相关产物 路径。区分当前执行的测试与已验证的现有产物。 绝不将静态预测与实测变异计数/评分结合。 最终 PR head 的证据不能证明每个中间 commit; 应针对所请求的 commit,单独相对于其 parent 进行评估。证据缺失并不自动等于已证明存在缺陷,也不应自动创建 inline review thread。发布应遵循调用方现有的授权门禁;不要仅为存储报告而创建新的 GitHub 对象, 也不要声称指令会强制执行 hooks、CI 或发布。
Validation
验证
- The evidence identifies the exact change, independent oracle, test/data case, and meaningful buildable counterfactual.
- A claimed bug-fix proof fails for the actual old defect and passes with the fix; a feature proof detects absent/wrong behavior rather than absent symbols.
- Executed claims have attributable outcomes, commands/exits, counts, and matching artifacts/consumer provenance; unavailable details stay explicit.
- Static predictions, survival, equivalence, preservation, and missing evidence remain distinct. Existing tests are credited without demanding redundant ones.
- Scope, read-only restrictions, clean variant outputs, parallel safety, and owned-resource cleanup were preserved.
- 证据明确标识准确变更、独立的 oracle、测试/数据用例, 以及有意义且可构建的反事实。
- 声称的缺陷修复证明会因真实的旧缺陷而失败,并在应用 修复后通过;新功能证明检测的是行为缺失/错误,而不是缺少符号。
- 执行声明具有可归因的结果、命令/退出状态、计数,以及匹配的产物/消费者来源;无法获取的细节应明确说明。
- 静态预测、survival、等价性、契约保持和证据缺失保持为不同概念。 现有测试应获得认可,不应要求重复测试。
- 始终遵守范围、只读限制、各变体输出清洁且相互隔离、并行安全 和自有资源清理要求。