Artifact Evaluation Prep

• a venue requires or offers artifact evaluation, reproducibility badges, or artifact appendices
• the user needs reviewer-facing install, quickstart, demo, or reproduction instructions
• a camera-ready or accepted paper needs an artifact package handoff
• code, data, checkpoints, models, Docker images, or external services must be packaged
• runtime, hardware, random seeds, expected outputs, or troubleshooting notes need to be made explicit
• claims in the paper need to be mapped to runnable scripts or released artifacts

release-code

• camera-ready-finalizer

  to recover accepted-paper obligations and final claim/evidence state

• release-code

  to prepare public repository hygiene after artifact obligations are clear

• reproducibility-audit

  when environment, data, or execution drift needs a broader audit

• run-experiment

  for generating or testing reproduction commands

• figure-results-review

  when artifact outputs must match paper figures or tables

• citation-audit

  when artifact metadata cites datasets, code, or prior artifacts

• research-project-memory

  when artifact status, blockers, and reviewer-facing instructions should persist

Skill Directory Layout

<installed-skill-dir>/
├── SKILL.md
└── references/
    ├── artifact-audit.md
    ├── memory-writeback.md
    ├── package-manifest.md
    ├── report-template.md
    └── reviewer-instructions.md

Progressive Loading

• Always read

  references/artifact-audit.md

  ,

  references/package-manifest.md

  , and

  references/reviewer-instructions.md

  .
• Read

  references/report-template.md

  before writing a saved artifact evaluation report.
• Read

  references/memory-writeback.md

  when the project has

  memory/

  , component

  .agent/

  folders, or the user asks for persistent state.
• If venue rules matter, verify current official artifact evaluation instructions before asserting deadlines, badge names, anonymity rules, upload fields, page limits, or required formats.

Core Principles

• Artifact evaluation is a reviewer workflow, not just a code dump.
• The artifact must reproduce the paper's important claims at an acceptable cost, or clearly document what it cannot reproduce.
• Prefer one reliable quickstart and one complete reproduction path over many fragile commands.
• Every command should state expected runtime, hardware, input, output, and success criteria.
• Package only redistributable data, checkpoints, and dependencies; document restricted assets precisely.
• Keep anonymity, licensing, and external-service assumptions explicit.
• Treat smoke tests as required. An untested instruction file is not an artifact package.

Step 1 - Recover Evaluation Context

• venue and artifact evaluation track, if known
• official artifact instructions, badge criteria, anonymity policy, and upload mechanism
• accepted or submitted paper, appendix, supplementary material, and checklist
• code repository, commit hash, branches, and worktrees
• datasets, checkpoints, pretrained models, generated outputs, and external dependencies
• hardware expectations: CPU/GPU type, memory, disk, runtime, network access
• paper claims, figures, tables, and experiments that the artifact should support
• constraints: private data, license limits, large files, cloud dependencies, nondeterminism, or reviewer time budget

Step 2 - Map Claims to Artifact Paths

• claim or result ID
• paper location
• script, notebook, config, or command that supports it
• input data or checkpoint
• expected output file, metric, table, or figure
• approximate runtime and hardware
• deterministic tolerance or expected variance
• reviewer priority: quickstart, core, optional, or not reproducible in package

Step 3 - Build the Artifact Manifest

references/package-manifest.md

• repository URL or archive path
• exact commit, tag, or checksum
• directory layout
• environment files and Docker images
• data and checkpoint locations
• reproduction scripts and configs
• expected generated outputs
• license and citation metadata
• known limitations and unsupported claims

ARTIFACT.md

REPRODUCE.md

docs/artifact_evaluation.md

Step 4 - Write Reviewer Instructions

references/reviewer-instructions.md

• setup commands
• quick smoke test under a short runtime budget
• core reproduction commands for main paper claims
• expected outputs and how to compare them with the paper
• troubleshooting for common failures
• hardware, storage, network, and time requirements
• contact policy or anonymous support channel if allowed
• limitations and optional extended runs

Step 5 - Smoke Test the Artifact

• environment creation or dependency resolution
• import or CLI sanity check
• quickstart command
• one representative data/checkpoint load
• one expected-output comparison

Step 6 - Handle Packaging Risks

• anonymization vs public release state
• licenses for code, data, pretrained weights, and third-party assets
• large-file strategy and checksums
• private paths, credentials, API keys, and machine-specific assumptions
• random seeds and nondeterminism
• version pinning and dependency conflicts
• reviewer time budget and failure recovery

release-code

reproducibility-audit

Step 7 - Write the Artifact Evaluation Report

references/report-template.md

docs/submission/artifact_evaluation_prep_YYYY-MM-DD.md

• readiness decision
• blocking issues
• claim-to-artifact map
• package manifest summary
• smoke-test status
• reviewer instruction status
• risks, limitations, and reviewer-facing caveats
• handoff to release, camera-ready, or memory

Step 8 - Write Back to Project Memory

references/memory-writeback.md

Final Sanity Check

• every important paper claim is either reproducible, smoke-tested, cached with explanation, or explicitly out of scope
• quickstart instructions have expected outputs and runtime
• hardware, data, checkpoints, licenses, and anonymity state are clear
• package paths and links are stable
• reviewer-facing failure modes are documented
• public-release and camera-ready obligations are routed
• project memory records artifact readiness and open blockers