test-research-code

Compare original and translation side by side

🇺🇸

Original

English

🇨🇳

Translation

Chinese

Test Research Code

研究代码测试

Brings a research code artifact to the state an artifact-evaluation committee (ACM AE, USENIX/OSDI/SOSP, SIGMOD ARI, ETAPS, NeurIPS code release) expects: it runs, it is deterministic enough to reproduce within tolerance, its environment is captured, and a stranger can follow the README to the paper's main result. This is the code-side counterpart to

preflight-check

for the PDF.

It is not generic software TDD. The goal is one reliable end-to-end "does it run and roughly reproduce" path, not exhaustive unit coverage. A focused smoke test that exercises the real pipeline on a tiny input is worth more than 100 mocked unit tests.

将研究code artifact调整至artifact评审委员会（ACM AE、USENIX/OSDI/SOSP、SIGMOD ARI、ETAPS、NeurIPS代码发布）期望的状态：代码可运行、具备足够的确定性（deterministic）以在误差范围内复现结果、环境已被捕获，且陌生用户可依照README复现论文的核心结果。这是针对PDF文档的

preflight-check

工具在代码层面的对应方案。

它并非通用软件的测试驱动开发（TDD）。目标是构建一条可靠的端到端“代码可运行且大致可复现”路径，而非全面的单元测试覆盖。针对真实流水线的小型输入执行聚焦式smoke test，价值远胜于100个模拟单元测试。

When to use

适用场景

The code is "works on my machine" and needs to become runnable by a stranger — no pinned env, no seeds, no obvious entrypoint, no sanity test.
The author asks to add a smoke/sanity test, pin seeds / make it deterministic, or capture/pin the environment before they package or hand it off.
An artifact-evaluation deadline is approaching and the code is not yet in testable shape — do this engineering first, then route to the siblings below.

Boundary (avoid overlap). This skill is the engineering step — tests, seeds, env capture. It does not own:

Packaging, badge taxonomy, archival DOI, anonymization →
```
prepare-artifacts
```
.
Whether the produced numbers actually match the paper's tables →
```
verify-results
```
.
General code cleanup/refactor →
```
refactor-research-code
```
. When the request is "get my code ready for the artifact track," start here for the run-ability gaps, then hand off; don't re-do their work.

代码处于“仅在本地可运行”状态，需要调整为陌生用户也能运行的状态——未固定环境、未设置种子、无明确入口点、无 sanity test。
作者要求在打包或移交前添加smoke/sanity test、固定种子/实现确定性，或捕获/固定环境。
artifact评审截止日期临近，但代码尚未具备可测试性——先完成这项工程工作，再交由后续相关流程处理。

边界（避免重叠）：该技能属于工程环节——测试、种子、环境捕获。它不负责：

打包、徽章分类、存档DOI、匿名化 →
```
prepare-artifacts
```
。
生成的数值是否与论文表格匹配 →
```
verify-results
```
。
通用代码清理/重构 →
```
refactor-research-code
```
。当需求是“让我的代码准备好提交artifact赛道”时，先在此解决可运行性问题，再移交后续流程；不要重复他人的工作。

Inputs

输入

A path to the research-code directory (the repo or a subfolder with training/eval scripts, etc.).
Optionally, whether review is double-blind (so a captured env / added test does not leak author identity).
Optionally
```
.paper-memory/profile.yml
```
for positioning (a
```
system
```
/
```
dataset
```
contribution leans harder on a reusable, well-tested artifact; an
```
empirical
```
one on deterministic re-runs).

研究代码目录的路径（仓库或包含训练/评估脚本等的子文件夹）。
可选：评审是否为双盲（确保捕获的环境/添加的测试不会泄露作者身份）。
可选：用于定位的
```
.paper-memory/profile.yml
```
文件（
```
system
```
/
```
dataset
```
类成果更侧重可复用、测试完善的artifact；
```
empirical
```
类成果更侧重确定性复现）。

Process

流程

Read memory first. Read
```
.paper-memory/lessons.md
```
(and
```
profile.yml
```
if present) so you don't re-flag what the author already fixed and you lead with their
```
recurring
```
habits. See
```
paper-memory-convention.md
```
.
Confirm scope. Ask whether review is double-blind (if not stated) so nothing added — a captured env, a test fixture, a notebook — leaks identity. If the author is targeting a specific artifact track, note that badge offerings and the separate artifact deadline must be re-verified live against the venue's current Call for Artifacts because they change per venue per year — but that resolution lives in
```
prepare-artifacts
```
. Here, just make the code testable; don't restate the badge taxonomy. Background context (badge families, the post-2020 Reproduced/Replicated swap) is in
```
references/artifact-standards.md
```
for awareness, not for asserting a current rule.
Audit the repo (deterministic). Run the bundled script — do not hand-grep:
```
python3 scripts/repro_check.py <code-dir>          # text report
python3 scripts/repro_check.py <code-dir> --json   # machine-readable
```
It reports six essentials — README, ENV (deps, pinned?), SEED, ENTRYPOINT, DATA, RESULTS_CMD — as OK / WARN / MISSING, plus whether any tests exist. This is the external, measurable signal the rest of the work is grounded in; do not substitute your own judgment of "looks reproducible". Exit code: 0 clean, 1 missing essential, 2 usage error.
Close the gaps, in this order (each maps to a script finding and to
```
references/repro-essentials.md
```
):
- ENV — write/repair the dependency manifest and pin exact versions.
```
pip freeze > requirements.txt
```
  , an
```
environment.yml
```
  , or a
```
Dockerfile
```
  that pins a base image. Unpinned deps are the single most common reason an evaluator cannot rebuild. Capture the OS/CUDA/hardware the results were produced on.
- SEED — set and record seeds for every RNG in play (
```
random
```
  ,
```
numpy
```
  , framework —
```
torch.manual_seed
```
  +
```
torch.use_deterministic_algorithms(True)
```
  ,
```
PYTHONHASHSEED
```
  ). Document residual nondeterminism (GPU atomics, data-loader workers) honestly rather than hiding it.
- ENTRYPOINT — provide one obvious command (
```
make reproduce
```
  ,
```
run.sh
```
  ,
```
python -m pkg
```
  , a
```
__main__
```
  guard) that runs the pipeline end to end.
- SMOKE/SANITY TEST — add a minimal test that runs the real pipeline on a tiny/synthetic input and asserts it completes and produces a sane shape/value. See the template in
```
references/smoke-tests.md
```
  . One end-to-end smoke test beats broad unit coverage here.
- DATA — give exact instructions to obtain inputs (download script, or "no external data"), with the source and a checksum where possible.
- RESULTS_CMD — put in the README a short table of reported results and, per result, the exact command that produces it (the ML Code Completeness item). Note expected runtime and that numbers should match within tolerance, not bit-for-bit.
Re-run
repro_check.py
until essentials are OK (or consciously WARN). Set an explicit stop condition — all essentials non-MISSING, or a hard cap of ~3 fix passes — rather than polishing open-endedly.
Hosting & anonymization (advise, never act).
- Archival hosting (Zenodo/SWHID, version-vs-concept DOI, the USENIX no-bare-GitHub rule) is owned by
```
prepare-artifacts
```
  . Do not resolve it here — once the code is testable, flag "next step: package + deposit via prepare-artifacts" and stop.
- Double-blind: if the repo is reviewed blind, anything you add (env file, fixture, notebook) must not reintroduce identity. Anonymization itself routes to
```
anonymize-paper
```
  /
```
prepare-artifacts
```
  ; flag it, don't do a half job inline.
Write the report and checklist to
```
paper-workspace/submission/artifact-repro-checklist.md
```
and append to
```
paper-workspace/INDEX.md
```
. Lead with what's done, then the ranked gap list, each tied to a
```
repro_check.py
```
finding and a concrete fix.

优先读取记忆文件。读取
```
.paper-memory/lessons.md
```
（若存在则同时读取
```
profile.yml
```
），避免重复标记作者已修复的问题，并遵循作者的
```
recurring
```
习惯。详见
```
paper-memory-convention.md
```
。
确认范围。若未说明，询问评审是否为双盲，确保添加的任何内容——捕获的环境、测试夹具、笔记本——不会泄露身份。若作者针对特定artifact赛道，需注意徽章设置和独立的artifact截止日期必须根据会议当前的Call for Artifacts实时确认，因为这些规则每年会随会议变化——但该问题由
```
prepare-artifacts
```
处理。在此仅需确保代码可测试；无需重复说明徽章分类。背景信息（徽章类别、2020年后Reproduced/Replicated术语变更）见
```
references/artifact-standards.md
```
，仅供参考，无需断言当前规则。
审核仓库（确保确定性）。运行捆绑脚本——不要手动搜索：
```
python3 scripts/repro_check.py <code-dir>          # 文本报告
python3 scripts/repro_check.py <code-dir> --json   # 机器可读格式
```
它会报告六项核心内容——README、ENV（依赖项是否固定？）、SEED、ENTRYPOINT、DATA、RESULTS_CMD——的状态为OK/WARN/MISSING，同时检查是否存在测试。这是后续工作的外部可衡量依据；不要用个人判断替代“看起来可复现”的结论。退出码：0表示无问题，1表示缺少核心内容，2表示使用错误。
按以下顺序填补漏洞（每项对应脚本发现及
```
references/repro-essentials.md
```
）：
- ENV —— 编写/修复依赖清单并固定精确版本。使用
```
pip freeze > requirements.txt
```
  、
```
environment.yml
```
  或固定基础镜像的
```
Dockerfile
```
  。未固定的依赖项是评审者无法重建环境的最常见原因。记录生成结果所使用的操作系统/CUDA/硬件信息。
- SEED —— 设置并记录所有随机数生成器（RNG）的种子（
```
random
```
  、
```
numpy
```
  、框架——如
```
torch.manual_seed
```
  +
```
torch.use_deterministic_algorithms(True)
```
  、
```
PYTHONHASHSEED
```
  ）。如实记录剩余的非确定性因素（GPU原子操作、数据加载器工作进程），而非隐藏它们。
- ENTRYPOINT —— 提供一个明确的命令（
```
make reproduce
```
  、
```
run.sh
```
  、
```
python -m pkg
```
  、
```
__main__
```
  guard），用于端到端运行流水线。
- SMOKE/SANITY TEST —— 添加一个最小化测试，针对小型/合成输入运行真实流水线，并断言其能完成运行且生成合理的结果形状/数值。详见
```
references/smoke-tests.md
```
  中的模板。在此场景下，一个端到端的smoke test胜过大量单元测试。
- DATA —— 提供获取输入的精确说明（下载脚本，或“无需外部数据”），尽可能包含来源和校验和。
- RESULTS_CMD —— 在README中添加一个简短的报告结果表格，针对每个结果提供生成它的精确命令（ML代码完整性项）。注明预期运行时间，以及数值应在误差范围内匹配，而非完全一致。
重新运行
repro_check.py
，直到核心内容状态为OK（或有意识地保留WARN）。设置明确的停止条件——所有核心内容无MISSING，或最多进行约3次修复迭代——而非无限期优化。
托管与匿名化（仅建议，不执行）：
- 存档托管（Zenodo/SWHID、版本DOI与概念DOI、USENIX禁止裸GitHub规则）由
  prepare-artifacts
  负责。在此不要处理该问题——代码具备可测试性后，标记“下一步：通过prepare-artifacts进行打包+存档”并停止。
- 双盲评审：若仓库采用盲审，添加的任何内容（环境文件、夹具、笔记本）不得重新引入身份信息。匿名化本身由
```
anonymize-paper
```
  /
```
prepare-artifacts
```
  处理；仅需标记该问题，不要在流程中半吊子处理。
撰写报告与检查清单，保存至
```
paper-workspace/submission/artifact-repro-checklist.md
```
并追加到
```
paper-workspace/INDEX.md
```
。首先说明已完成的工作，然后按优先级列出漏洞清单，每项关联
```
repro_check.py
```
的发现及具体修复方案。

Output

输出

A reviewable

paper-workspace/submission/artifact-repro-checklist.md

: the audit table (six essentials, OK/WARN/MISSING), the smoke-test you added or recommended, the env-capture command run, the seeds pinned, and the data/reproduce instructions — plus any actual files written into the author's repo (a

tests/test_smoke.py

, a pinned

requirements.txt

, a

Makefile

run.sh

). Every gap cites the script finding it came from. The checklist ends with a one-line handoff to

prepare-artifacts

(packaging/badge/DOI) and

verify-results

(do the numbers match) — it does not duplicate their work.

一份可评审的

paper-workspace/submission/artifact-repro-checklist.md

：包含审核表格（六项核心内容，状态为OK/WARN/MISSING）、添加或推荐的smoke test、运行的环境捕获命令、固定的种子、数据/复现说明——以及写入作者仓库的所有实际文件（如

tests/test_smoke.py

、固定版本的

requirements.txt

、

Makefile

run.sh

）。每个漏洞都需引用发现它的脚本结果。检查清单末尾需添加一行，移交至

prepare-artifacts

（打包/徽章/DOI）和

verify-results

（数值是否匹配）——不要重复它们的工作。

Adapt to your discipline

适配不同学科

This targets CS artifact tracks (ACM/USENIX/SIGMOD/ML). For other fields: swap the env-capture for your stack's lockfile (

renv.lock

for R,

Manifest.toml

for Julia,

conda

for bioinformatics), and swap the seed/test idioms for your framework's. Badge/venue mapping is

prepare-artifacts

's job, not this skill's — here, "testable + deterministic + env captured" is field-agnostic.

本工具针对计算机科学的artifact赛道（ACM/USENIX/SIGMOD/ML）。对于其他领域：将环境捕获方式替换为对应技术栈的锁定文件（R语言用

renv.lock

，Julia用

Manifest.toml

，生物信息学用

conda

），并将种子/测试习惯替换为对应框架的方式。徽章/会议映射由

prepare-artifacts

负责，而非本技能——在此，“可测试+确定性+环境已捕获”是跨领域通用的要求。

Guardrails

约束规则

Never run anything destructive, and never run untrusted experiment code to "prove" it works without the author's say-so.
```
repro_check.py
```
is static and read-only by design (no execution, no network). Running the artifact end-to-end is the author's call and may need a sandbox/GPU.
A passing checklist is necessary, not sufficient. Do not tell the author the artifact "will earn the badge" or "is reproducible" — those are the committee's findings against the live CFA, and "reproduce" everywhere means within a tolerance that doesn't change the paper's claims, never bit-exact. Use necessary-not-sufficient language.
Don't fabricate results or commands. The reproduce command must be one that actually exists in the repo; if it doesn't run, say so.
Anonymization-aware: for double-blind review, never expose author identity through the repo, commit history, or notebook metadata.
It checks, scaffolds, and explains; it does not submit the artifact or mint a DOI on the author's behalf.
```
references/
```
files stay one level deep; keep this file under 500 lines.

绝不运行任何破坏性代码，未经作者许可绝不运行不可信的实验代码以“证明”其可运行。
```
repro_check.py
```
设计为静态只读工具（不执行代码，不联网）。端到端运行artifact需由作者决定，且可能需要沙箱/GPU环境。
检查清单通过是必要条件，但非充分条件。不要告知作者该artifact“将获得徽章”或“具备可复现性”——这些是评审委员会根据当前Call for Artifacts得出的结论，且“复现”在所有场景下均指在不改变论文结论的误差范围内匹配，而非完全一致。需使用必要非充分的表述。
不要编造结果或命令。复现命令必须是仓库中实际存在的命令；若无法运行，如实说明。
注意匿名化：对于双盲评审，绝不要通过仓库、提交历史或笔记本元数据泄露作者身份。
本工具仅负责检查、搭建框架和说明；不负责提交artifact或代表作者生成DOI。
```
references/
```
目录下的文件保持一级深度；本文件长度控制在500行以内。

Memory

记忆机制

Uses the shared

.paper-memory/

convention (full spec:

paper-memory-convention.md

At start: read
```
lessons.md
```
to skip already-fixed gaps and lead with
```
recurring
```
habits (e.g. "you tend to ship unpinned deps — checking that first"); read
```
profile.yml
```
for contribution type so the badge target is right.

At end: append one dated entry per recurring gap, in the canonical format, via

reflect-and-improve

reflect_log.py

python3 ../reflect-and-improve/scripts/reflect_log.py append \
    --memory .paper-memory --skill test-research-code --scope recurring \
    --issue "experiment scripts ship without seed setting" \
    --rec "set random/numpy/framework seeds + PYTHONHASHSEED every artifact"

Create
```
.paper-memory/
```
on demand; offer to add it to
```
.gitignore
```
; local-only, never uploaded.

遵循共享的

.paper-memory/

约定（完整规范：

paper-memory-convention.md

）。

开始时：读取
```
lessons.md
```
以跳过已修复的漏洞，并遵循
```
recurring
```
习惯（例如：“您通常会发布未固定版本的依赖项——优先检查此项”）；读取
```
profile.yml
```
以了解成果类型，确保徽章目标正确。

结束时：通过

reflect-and-improve

的

reflect_log.py

，按标准格式为每个重复出现的漏洞添加一条带日期的记录：

python3 ../reflect-and-improve/scripts/reflect_log.py append \
    --memory .paper-memory --skill test-research-code --scope recurring \
    --issue "experiment scripts ship without seed setting" \
    --rec "set random/numpy/framework seeds + PYTHONHASHSEED every artifact"

按需创建
```
.paper-memory/
```
目录；建议将其添加到
```
.gitignore
```
；仅本地存储，绝不上传。