verify-results

Compare original and translation side by side

🇺🇸

Original

English

🇨🇳

Translation

Chinese

Verify Results

验证结果

Close the loop between what the paper claims and what the artifact produces. This skill helps the author confirm their reported numbers reproduce: it locates the experiment code, helps stand up a clean/sandboxed run, runs the artifact's own tests, and does a consistency audit — comparing the metrics the run produces against the paper's tables and claims, within a tolerance that does not change the paper's conclusions. It reports mismatches (paper says X, code produces Y) and missing reproduction steps, and audits the artifact against current reproducibility-badge expectations.

It is a copilot: it sets up and guides, and the author runs anything heavy (training, long evals) in their own environment. It never fabricates a number, never executes destructive commands, and never claims a result was independently reproduced — a clean audit means consistent, not reproduced.

打通论文声明与工件产出之间的闭环。该技能帮助作者确认其报告的数据可复现：它定位实验代码，协助搭建干净/沙盒化运行环境，运行工件自带的测试，并执行一致性审计——将运行产生的指标与论文表格及声明进行对比，对比容差不会改变论文结论。它会报告不匹配情况（论文显示X，代码产出Y）和缺失的复现步骤，并根据当前可复现性badge要求对工件进行审计。

它是一个副驾驶：负责搭建和引导，作者在自己的环境中运行计算密集型任务（训练、长时间评估）。它绝不会编造数据，绝不会执行破坏性命令，也绝不会声称结果已独立复现——干净的审计仅意味着一致，而非复现。

When to use

使用场景

"Do my results reproduce?" / "Does my code match the paper's tables?"
"Check my reproducibility" / "verify my experiments" / "reproduce my numbers".
Prepping an artifact for an evaluation track (ACM AE, USENIX, OSDI, SOSP, SIGMOD ARI, ETAPS, NeurIPS/ICML/ACL reproducibility).
Filling a reproducibility checklist (NeurIPS Paper Checklist, ACL Responsible NLP, ML Code Completeness) and wanting an honest read on each item.
After a results table changes and you need to confirm the code still produces it.

“我的结果能复现吗？” / “我的代码与论文表格匹配吗？”
“检查我的可复现性” / “验证我的实验” / “复现我的数据”。
为评估赛道准备工件（ACM AE、USENIX、OSDI、SOSP、SIGMOD ARI、ETAPS、NeurIPS/ICML/ACL可复现性赛道）。
填写可复现性清单（NeurIPS论文清单、ACL负责任NLP、ML代码完整性清单）并希望对每项内容进行客观评估。
修改结果表格后，确认代码仍能产出对应结果。

Inputs

输入

The artifact / experiment code (a directory; a repo URL the author has cloned locally — this skill reads local files, it does not clone for you).
The paper
```
.tex
```
whose tables/claims are being checked (or the specific
```
\input
```
file that holds the results table).
The target venue's artifact track, if any — its current Call for Artifacts decides which badges exist and what hosting they require.
Optionally, a metrics file from a prior run (JSON/CSV) to compare without re-running.

工件/实验代码（一个目录；作者已本地克隆的仓库URL——该技能读取本地文件，不会帮你克隆）。
待检查表格/声明的论文
```
.tex
```
文件（或包含结果表格的特定
```
\input
```
文件）。
目标会议的artifact赛道（如有）——其当前的Artifact征集要求（Call for Artifacts）决定了可用的badge类型及所需的托管方式。
可选：之前运行生成的指标文件（JSON/CSV格式），用于无需重新运行的对比。

Process

流程

This skill follows plan → set up → run (author) → audit, with the verification step grounded in external, measurable signals (test pass/fail, a numeric diff against a file the run produced) — never the model's own judgment that the numbers "look right".

Locate the experiment code and the claims. Confirm where the code lives and which paper tables/claims it is supposed to produce. Extract the paper's reported numbers into a reviewable claims ledger:
```
python3 scripts/extract_claims.py paper.tex --ledger claims.json
```
The ledger is a starting point, not ground truth — walk it with the author: drop spurious numbers (years, citation counts, the
```
top-1=1
```
from a
```
\\
```
row), fix metric labels, mark each kept claim
```
confirmed
```
. The author is the author.
Audit the artifact for completeness and badge-readiness.
```
python3 scripts/audit_repo.py path/to/artifact --blind <single|double|none>
```
This inventories the repo against the ML Code Completeness Checklist (dependencies, training code, evaluation code, pretrained models, a README with a results table + exact reproduce command), checks for the artifact's own tests, flags missing reproduction steps, and warns when the only hosting is a GitHub/personal URL (badge tracks want an archival DOI — Zenodo/FigShare/Dryad/Software Heritage). It runs nothing. Under
```
--blind double
```
it also scans the README for de-anonymizing emails/URLs.
This is a fast pre-comparison gate, not the deep version. Don't re-do work the sibling skills own: making the code run-ready/deterministic and the repro-essentials audit belong to
```
test-research-code
```
; packaging, the badge taxonomy, and the archival DOI belong to
```
prepare-artifacts
```
; the deep double-blind sweep belongs to
```
anonymize-paper
```
/
```
refactor-research-code
```
. The unique job of this skill is the consistency audit (step 5) — does the run's output match the paper's tables. Use
```
audit_repo.py
```
only to confirm there is enough to run before comparing, then hand deep gaps to the owner skill.
Re-verify the badge rules against the live Call for Artifacts — mandatory. Badge offerings change per venue, per year (e.g. one cycle a venue offers all three badges; another, only Artifacts Available). The terms Reproduced vs Replicated were swapped by ACM after 2020-05-14 — pre-2020 papers use the inverse meanings. Do not state any badge requirement, hosting rule, or deadline from memory: fetch the venue's current CFA and confirm it with a source URL and access date. Reproducibility standards and the badge taxonomy are in references/repro-standards.md — treat it as a map of what to verify, not as current truth.
Set up a clean, sandboxed run — then hand the author the commands. A reproduction must run from a pinned, isolated environment, not the author's polluted shell. Help build the recipe (fresh venv/conda/container from the dependency spec; seeds fixed; the exact command from the README), but the author runs anything heavy. See references/sandbox-and-run.md. First have the author run the artifact's own tests (
```
pytest
```
,
```
make test
```
, the repo's harness) — a concrete pass/fail gate before any metric comparison. Never run destructive commands; never auto-install into the author's base environment.
Consistency audit: compare produced metrics to the paper. Point the run's output (a metrics JSON/CSV the author generated) at the confirmed ledger:
```
python3 scripts/compare_metrics.py --ledger claims.json --metrics run.json \
    --rel-tol 0.01 --abs-tol 0.005 [--map test_acc=c1 ...]
```
It reports MATCH / MISMATCH / MISSING per claim with a tolerance that does not change the paper's claim — never bit-exact (ACM, SIGMOD ARI, and ETAPS all require only agreement within tolerance / "similar behavior"). Tune
```
--rel-tol
```
/
```
--abs-tol
```
to the metric's scale and use
```
--map
```
when names differ. A
```
metric/no-produced-value
```
is a missing repro step (the paper reports it; the run didn't emit it).
Decide the verification outcome with explicit stop conditions. Map each compared claim to: match (consistent within tolerance), mismatch (paper says X, code produces Y — reconcile: stale table? wrong seed? different split? selective reporting?), or unverified (could not run / metric not emitted — say so, never paper over it). Do not loop indefinitely: stop when all confirmed claims are match-or-explained, or escalate to the author when a mismatch needs a judgment call (which number is right). Escalation is a feature, not a failure (working-principle #4).
Write the reproduction report to
```
paper-workspace/review/reproduction-report.md
```
and append a line to
```
paper-workspace/INDEX.md
```
. Lead with the verdict (N of M claims consistent), then the mismatch table (claim, paper value, produced value, |diff|, likely cause), the artifact-completeness checklist with each item's status, missing repro steps, and badge-readiness per the live CFA. State plainly what was and was not actually run.

该技能遵循规划→搭建→运行（作者执行）→审计的流程，验证步骤基于外部可测量信号（测试通过/失败、与运行生成文件的数值差异）——绝不依赖模型自身对数据“看起来正确”的判断。

定位实验代码与声明。确认代码位置及其应产出的论文表格/声明。将论文中报告的数据提取为可审查的声明台账：
```
python3 scripts/extract_claims.py paper.tex --ledger claims.json
```
台账是一个起点，而非真相——与作者一起梳理：剔除无关数据（年份、引用次数、
```
\
```
行中的
```
top-1=1
```
），修正指标标签，标记每个保留的声明为
```
confirmed
```
。作者拥有最终决定权。
审计工件的完整性与badge就绪状态。
```
python3 scripts/audit_repo.py path/to/artifact --blind <single|double|none>
```
该脚本根据ML代码完整性清单（依赖项、训练代码、评估代码、预训练模型、包含结果表格+精确复现命令的README）对仓库进行盘点，检查工件的自带测试，标记缺失的复现步骤，并警告仅使用GitHub/个人URL托管的情况（badge赛道要求归档DOI——Zenodo/FigShare/Dryad/Software Heritage）。它不会运行任何代码。在
```
--blind double
```
模式下，还会扫描README中的去匿名化邮箱/URL。
这是一个快速预对比关卡，而非深度版本。不要重复其他兄弟技能的工作：使代码可运行/确定性，以及复现要素审计属于
```
test-research-code
```
；打包、badge分类体系、归档DOI属于
```
prepare-artifacts
```
；深度双盲扫描属于
```
anonymize-paper
```
/
```
refactor-research-code
```
。本技能的独特任务是一致性审计（步骤5）——运行输出是否与论文表格匹配。仅使用
```
audit_repo.py
```
确认对比前有足够的可运行内容，然后将深度问题移交对应技能处理。
根据最新的Artifact征集要求重新验证badge规则——强制要求。 Badge提供内容因会议、年份而异（例如，某一周期会议提供全部三种badge；另一周期仅提供Artifacts Available）。Reproduced与Replicated的定义在2020-05-14之后被ACM互换——2020年前的论文使用相反含义。切勿凭记忆陈述任何badge要求、托管规则或截止日期：获取会议当前的CFA（Call for Artifacts）并通过源URL和访问日期进行确认。可复现性标准和badge分类体系见references/repro-standards.md——将其视为验证内容的参考，而非当前事实。
搭建干净的沙盒化运行环境——然后将命令交给作者。复现必须从固定的隔离环境运行，而非作者已受污染的shell。协助构建运行方案（从依赖规范创建全新venv/conda/容器；固定随机种子；使用README中的精确命令），但计算密集型任务由作者执行。详见references/sandbox-and-run.md。首先让作者运行工件自带的测试（
```
pytest
```
、
```
make test
```
、仓库的测试框架）——这是进行任何指标对比前的明确通过/失败关卡。绝不运行破坏性命令；绝不自动安装到作者的基础环境中。
一致性审计：对比产出指标与论文。将运行输出（作者生成的指标JSON/CSV文件）与已确认的台账关联：
```
python3 scripts/compare_metrics.py --ledger claims.json --metrics run.json \
    --rel-tol 0.01 --abs-tol 0.005 [--map test_acc=c1 ...]
```
它会针对每个声明报告匹配/不匹配/缺失状态，使用的容差不会改变论文的结论——绝不要求完全一致（ACM、SIGMOD ARI和ETAPS均仅要求在容差内一致/“行为相似”）。根据指标规模调整
```
--rel-tol
```
/
```
--abs-tol
```
，当名称不同时使用
```
--map
```
。
```
metric/no-produced-value
```
表示缺失复现步骤（论文报告了该指标，但运行未输出）。
通过明确的终止条件决定验证结果。将每个对比的声明映射为：匹配（容差内一致）、不匹配（论文显示X，代码产出Y——需调和：表格过时？种子错误？数据集拆分不同？选择性报告？），或未验证（无法运行/未输出指标——如实报告，绝不掩盖）。不要无限循环：当所有已确认的声明均匹配或已解释，或当不匹配情况需要作者判断（哪个数据正确）时升级给作者。升级是功能，而非失败（工作原则#4）。
撰写复现报告至
```
paper-workspace/review/reproduction-report.md
```
，并在
```
paper-workspace/INDEX.md
```
中添加一行。开头给出结论（M项声明中有N项一致），然后是不匹配表格（声明、论文值、产出值、|差异|、可能原因）、工件完整性清单及各项状态、缺失的复现步骤、以及根据最新CFA的badge就绪状态（含源链接和日期）。明确说明实际运行和未运行的内容。

Output

输出

reproduction-report.md

: verdict (consistent claims / total) → mismatch table (paper vs produced, with diffs and suspected cause) → artifact completeness checklist (5 items + tests + hosting) → missing reproduction steps → badge-readiness against the live CFA (with source links and dates). Plus the machine-readable

claims.json

ledger and the

--json

outputs if requested.

一份

reproduction-report.md

：结论（一致声明数/总声明数）→不匹配表格（论文值vs产出值，含差异和疑似原因）→工件完整性清单（5项+测试+托管）→缺失的复现步骤→根据最新CFA的badge就绪状态（含源链接和日期）。此外还有机器可读的

claims.json

台账，以及若有请求的

--json

输出。

Adapt to your discipline

适配你的学科

The metric heuristics target ML/systems papers (accuracy, F1, BLEU, latency, speedup...). For other fields, the ledger is just

{metric, value}

records — hand-author it for any quantitative claim (effect sizes, p-values, runtimes) and

compare_metrics.py

still does the tolerance-aware audit. Non-code artifacts (datasets, proofs) use steps 2–3 only.

指标启发式方法针对ML/系统论文（准确率、F1、BLEU、延迟、加速比……）。对于其他领域，台账仅为

{metric, value}

记录——手动编写任何定量声明（效应量、p值、运行时间），

compare_metrics.py

仍会执行容差感知审计。非代码工件（数据集、证明）仅使用步骤2–3。

Related skills (don't duplicate them)

Need	Owner skill
Make the code run-ready/deterministic; repro-essentials audit	`test-research-code`
Packaging, badge taxonomy, archival DOI, artifact appendix	`prepare-artifacts`
Release cleanup of a research repo	`refactor-research-code`
Deep, reversible double-blind sweep	`anonymize-paper`
Each written claim traces to evidence (not just numbers)	`verify-claims`

需求	负责技能
使代码可运行/确定性；复现要素审计	`test-research-code`
打包、badge分类体系、归档DOI、工件附录	`prepare-artifacts`
研究仓库的发布清理	`refactor-research-code`
深度可逆双盲扫描	`anonymize-paper`
每个书面声明可追溯至证据（不仅是数据）	`verify-claims`

Guardrails

防护规则

Consistency is not reproduction. A clean audit says the produced numbers match the paper within tolerance — it does not mean the result was independently reproduced or replicated. Never claim a badge is earned; that is a committee's call against the live CFA.
Never fabricate a number, a metric, or a "passing" run. If a run did not happen or a metric was not emitted, report it as unverified — do not infer it.
Don't trust the model's own read of correctness. The verification signal is external: a test exit code, a numeric diff against a file the author produced. Self-reflection validates hallucinations — do not use it as the gate.
Run nothing heavy or destructive. The author runs training/long evals in their own sandbox; this skill sets up and guides. No installs into the base env, no
```
rm
```
, no network side effects.
Anonymization-aware. Under double-blind, flag identifying content in the artifact and prefer an anonymized mirror; never expose the author's identity.
Never submit the paper or the artifact to any system on the author's behalf.

一致性不等同于复现。干净的审计仅表明产出数据在容差内与论文匹配——绝不意味着结果已独立复现或重复验证。绝不声称已获得badge；这是委员会根据最新CFA做出的决定。
绝不编造数据、指标或“通过”的运行结果。若未运行或未输出指标，报告为未验证——绝不推断。
不要信任模型自身对正确性的判断。验证信号是外部的：测试退出码、与作者生成文件的数值差异。自我反思会验证幻觉——绝不将其作为关卡。
不运行计算密集型或破坏性命令。作者在自己的沙盒中运行训练/长时间评估；本技能负责搭建和引导。不安装到基础环境，不执行
```
rm
```
，不产生网络副作用。
支持匿名化。在双盲模式下，标记工件中的识别内容，优先使用匿名镜像；绝不暴露作者身份。
绝不代表作者提交论文或工件至任何系统。

Memory

记忆机制

Uses the shared

.paper-memory/

convention (full spec:

paper-memory-convention.md

At start: read
```
.paper-memory/lessons.md
```
(and
```
profile.yml
```
for the contribution type — a
```
dataset
```
/
```
system
```
paper is judged on the artifact more heavily). Lead with any
```
recurring
```
repro habits (e.g. "tables drift from the code between drafts", "unpinned dependencies").
At end: append each finding worth remembering as one dated entry in the shared format
```
- [YYYY-MM-DD] (verify-results | <scope>) issue -> recommendation
```
(use
```
reflect-and-improve
```
's
```
reflect_log.py append
```
, which dedupes and dates). Tag a cross-paper habit
```
recurring
```
, a one-off
```
this-paper
```
.
Create
```
.paper-memory/
```
on demand and offer to add it to
```
.gitignore
```
. It is local-only; never upload it or copy it into this repo.

使用共享的

.paper-memory/

约定（完整规范：

paper-memory-convention.md

）。

开始时：读取
```
.paper-memory/lessons.md
```
（以及
```
profile.yml
```
以了解贡献类型——
```
dataset
```
/
```
system
```
论文对工件的评判更严格）。首先处理任何
```
recurring
```
复现习惯（例如，“草稿间表格与代码不一致”、“未固定依赖项”）。
结束时：将每个值得记录的发现以共享格式添加为一条带日期的条目：
```
- [YYYY-MM-DD] (verify-results | <scope>) issue -> recommendation
```
（使用
```
reflect-and-improve
```
的
```
reflect_log.py append
```
，它会去重并添加日期）。将跨论文的习惯标记为
```
recurring
```
，一次性问题标记为
```
this-paper
```
。
按需创建
```
.paper-memory/
```
并建议将其添加到
```
.gitignore
```
。它仅保存在本地；绝不上传或复制到本仓库。