maintain-formal-verification-skill

Compare original and translation side by side

🇺🇸

Original

English
🇨🇳

Translation

Chinese

Maintain a formal verification skill

维护 formal verification skill

A formal check proves a property of what it describes. When the code changes and the verification does not, the check can keep passing while it describes code that no longer exists. That pass is the dangerous case, because it looks like a proof. This skill is the upkeep loop for a verify skill that the create-formal-verification-skill skill generated, or for any project-local skill that runs a model checker, prover, or code-level verifier against named code. The project-local skill handles a change that an agent makes while it uses that skill. This skill handles the changes that skipped it: merges, edits made without the skill, and drift that builds up across many commits.
formal check 会证明其所描述对象的某项 property。当代码发生变化而 verification 未同步时,check 可能继续通过,但它所描述的代码实际上已经不存在。这种通过尤其危险,因为它看起来像是一项证明。这个 skill 是维护 verify skill 的 upkeep loop;既适用于 create-formal-verification-skill 生成的 verify skill,也适用于任何针对指定代码运行 model checker、prover 或 code-level verifier 的 project-local skill。project-local skill 负责 Agent 使用该 skill 期间发生的代码变更;这个 skill 则负责绕过该 skill 的变更,包括 merges、未通过该 skill 进行的编辑,以及经过多个 commits 逐渐累积的漂移。

Outcomes

Outcomes

Give each check one outcome, and say which:
  • in sync: no change since the last sync alters what the check describes, and the check passes.
  • updated: the code changed, the verification now describes the new code, and the check passes.
  • bug: the verification describes the current code, and the code breaks the property. This is a finding about the code. It is not drift.
  • blocked: the check could not be brought back in line or could not run. Say exactly why.
A check can be both updated and bug: the update makes the verification describe the code again, and then the property fails on it.
为每项 check 指定一个 outcome,并明确说明:
  • in sync:自上次同步以来,没有任何变更影响 check 所描述的内容,并且 check 通过。
  • updated:代码已发生变化,verification 现在描述了新代码,并且 check 通过。
  • bug:verification 描述的是当前代码,但代码违反了 property。这是关于代码的发现,并非漂移。
  • blocked:无法使 check 重新保持一致,或者无法运行它。必须准确说明原因。
一项 check 可以同时是 updated 和 bug:更新使 verification 再次与代码保持一致,随后该 property 在代码上验证失败。

Edit scope

Edit scope

Edit only the verification: models, harnesses, property files, and the verify skill itself. Never edit product code during a run, unless the user asks you to. When a property fails on the current code, do not weaken the property or the model to make it pass. Report the failure.
只能编辑 verification,包括 models、harnesses、property files 和 verify skill 本身。除非用户明确要求,否则运行期间绝不能编辑产品代码。当 property 在当前代码上失败时,不要通过削弱 property 或 model 来让它通过。应报告该失败。

Pass

Pass

  1. Find the checks. A verify skill has no fixed name, so find it by what it does: a project skill that runs a model checker, a prover, or another verifier against named code. Read each one you find, or the one the user named. For each check, collect the property, the code it covers, the bounds and assumptions, the command that runs it, and the commit where it was last confirmed to describe the code. If no verify skill exists, stop and point at
    /create-formal-verification-skill
    . If the skill does not record a sync commit, use the last commit that changed the verification files, and say that you did.
  2. Run it as it is. Run each check once before you change anything, and record the result. A pass here does not tell you that the code is correct. It tells you that the old description still holds. A failure here can be drift or a bug. Do not decide which until step 5.
  3. Read every change to covered code. Diff the covered files from the sync commit to the current commit, and read each commit message. Also look for code that moved: a covered function that is renamed, split, or moved to another file is still covered code. Also look for new code that touches the state that a check guards, such as a new function that writes the same fields. For each change, decide which of these it is:
    • No effect. The change does not alter anything the verification describes, such as formatting, comments, or code paths that the verification does not cover. Check this against the model or harness, not against the commit message.
    • The verification no longer describes the code. A step that was one indivisible action is now two, a branch was added or removed, a check moved, a new path reaches the guarded state, or a value now has a different range. The model or harness must change.
    • The promise changed on purpose. The change alters what the code promises, and evidence in the change says that this is intended: the doc comment, the tests, the changelog, or the commit message. The property must change, and you must cite that evidence.
    • The promise may be broken. The code changed, and no evidence says that the promise changed. Treat a failure that follows as a bug candidate.
  4. Update from the code, not from the result. Change the model or harness so that each changed part again points to code that a reviewer can open, and keep one indivisible step in the code as one step in the verification. Write the update from the code before you rerun, and do not shape it by what makes the check pass. Change a property only for a promise that changed on purpose. Write down the old wording, the new wording, and the evidence. Keep the bounds unless the change makes them wrong. If you must reduce a bound, say so in the report.
  5. Rerun and judge each result.
    • Pass: the check can now pass on a description that has lost the ability to fail. Break the new code the way the property guards against, such as removing a new check or reordering a new step, and confirm that the check catches it. Then drop the break.
    • Fail: replay the counterexample against the real code, step by step. If a step cannot happen in the code, the verification is wrong. Fix the verification, and say what was wrong. If every step can happen, it is a bug. Name the commit that introduced it, tell the counterexample as a sequence of steps in the code's own words, and state the harm in one sentence. If a cheap test or reproduction against the real code shows the bug, add it to the report as evidence. Do not commit it to the product code.
  6. Update the verify skill. Update the code that each check covers, the commands, the bounds, the assumptions, and the sync commit. Set the sync commit to the commit that you checked. Do not move the sync commit for a blocked check, and say so in the report. Then follow the verify skill once as the next agent would, so that you know its steps still run.
  1. 查找 checks。 verify skill 没有固定名称,因此应根据其行为查找:寻找运行 model checker、prover 或其他 verifier 来检查指定代码的 project skill。读取找到的每个 skill,或者读取用户指定的 skill。对于每项 check,收集 property、它覆盖的代码、bounds 和 assumptions、运行命令,以及最后一次确认它与代码一致时所在的 commit。如果不存在 verify skill,则停止操作并指向
    /create-formal-verification-skill
    。如果该 skill 没有记录 sync commit,就使用最后一次修改 verification files 的 commit,并说明这一选择。
  2. 按当前状态运行。 在进行任何修改前,先原样运行每项 check,并记录结果。此处的通过并不能说明代码是正确的,只能说明旧的描述仍然成立。此处的失败可能是漂移,也可能是 bug。在第 5 步之前不要决定属于哪一种。
  3. 阅读覆盖代码的所有变更。 对比 sync commit 与当前 commit 之间被覆盖文件的差异,并阅读每条 commit message。还要查找发生了移动的代码:被重命名、拆分或移动到其他文件的受覆盖函数仍属于受覆盖代码。还要查找触及 check 所保护状态的新代码,例如写入相同字段的新函数。针对每项变更,判断它属于以下哪一种:
    • No effect。 变更没有影响 verification 所描述的任何内容,例如格式调整、注释修改,或 verification 未覆盖的代码路径。应根据 model 或 harness 进行判断,而不是依据 commit message。
    • The verification no longer describes the code。 原本一个不可分割的操作现在变成两个操作,增加或删除了一个 branch,check 发生移动,新增路径可以到达受保护状态,或者某个值现在具有不同的范围。此时必须修改 model 或 harness。
    • The promise changed on purpose。 变更改变了代码所承诺的行为,并且变更中的证据表明这是有意为之,例如 doc comment、tests、changelog 或 commit message。此时必须修改 property,并引用相关证据。
    • The promise may be broken。 代码发生了变化,但没有任何证据表明相应承诺发生了改变。随后出现的失败应视为 bug 候选。
  4. 根据代码而不是结果进行更新。 修改 model 或 harness,使每个发生变化的部分再次指向 reviewer 能够直接查看的代码,并在 verification 中将代码里一个不可分割的 step 保留为一个 step。应在重新运行之前根据代码完成更新,不要根据什么修改能让 check 通过来决定如何修改。只有在 promise 有意改变时才修改 property。记录旧措辞、新措辞和相关证据。除非变更导致原有 bounds 失效,否则应保留这些 bounds。如果必须缩小某个 bound,请在报告中说明。
  5. 重新运行并判断每项结果。
    • Pass: check 现在能够在一个已经失去失败能力的描述上通过。按照 property 所防范的方式破坏新代码,例如移除新增的 check 或重新排序新增的 step,确认 check 能够发现该问题。随后撤销这项破坏性改动。
    • Fail: 使用真实代码逐步重放 counterexample。如果其中某个步骤不可能在代码中发生,则说明 verification 有误。修复 verification,并说明错误所在。如果每个步骤都可能发生,这就是一个 bug。指出引入该问题的 commit,使用代码本身的术语将 counterexample 描述为一系列步骤,并用一句话说明其危害。如果能够通过成本较低的 test 或针对真实代码的复现证明该 bug,可将其作为证据加入报告。不要将这类内容提交到产品代码中。
  6. 更新 verify skill。 更新 verify skill 中关于每项 check 所覆盖代码的信息、命令、bounds、assumptions 和 sync commit。将 sync commit 设置为你所检查的 commit。对于 blocked 的 check,不要移动其 sync commit,并在报告中说明。随后按照下一个 Agent 的使用方式完整执行一次 verify skill,以确认其中的步骤仍然可以运行。

Report

Report

Tell the user, for each check:
  • the outcome, and the commits that you read;
  • what changed in the verification and why, with a short reason for each change: the code changed shape, or the promise changed on purpose (with the evidence);
  • the rerun result, and the break that you used to show the updated check can still fail;
  • each bug, in plain words: the counterexample as steps in the code's words, the harm, and the commit that introduced it;
  • what the verification still does not cover, including new code that it should cover and does not.
Say outright which failures were drift in the verification and which were bugs in the code. For a check that passed before you updated it and fails after, say that the old pass was on a stale description.
Put the verification updates in a change of their own, separate from any product fix.
针对每项 check,向用户说明:
  • outcome,以及你阅读过的 commits;
  • verification 中变更了什么、为什么变更,并为每项变更给出一个简短理由:代码形态发生了变化,或者 promise 有意改变,并附上相应证据;
  • 重新运行的结果,以及你用于证明更新后的 check 仍然能够失败的破坏性改动;
  • 用通俗语言说明每个 bug:按照代码本身的术语列出 counterexample 的各个步骤、说明危害,并指出引入该问题的 commit;
  • verification 仍未覆盖的内容,包括本应覆盖但目前没有覆盖的新代码。
明确指出哪些失败属于 verification 漂移,哪些失败属于代码 bug。如果某项 check 在更新前通过、更新后失败,请说明此前的通过是基于过时的描述。
将 verification 更新放入单独的变更中,不要与任何产品修复混在一起。