benchmark-paper

Compare original and translation side by side

🇺🇸

Original

English

🇨🇳

Translation

Chinese

Benchmark Paper

论文基准测试工具

Produces a venue-fit scorecard: how closely a draft matches the measurable patterns of recent award-winning and top-cited papers at the target venue. This is a conformance gauge to help an author find gaps before submitting — it is not a prediction of acceptance, a best-paper forecast, or a judgment of scientific merit.

Pairs naturally with

simulate-reviewers

(which red-teams content quality) and

study-exemplars

(which it calls to build the comparison corpus). Run

preflight-check

first — a desk-reject defect makes any score moot.

生成会议适配度评分卡：评估论文草稿与目标会议近期获奖及高引用论文的可衡量模式的契合程度。这是一个一致性衡量指标，帮助作者在提交前找出不足——绝非对录用结果、最佳论文奖项的预测，也不是对科学价值的评判。

可与

simulate-reviewers

（针对内容质量进行严苛评审）和

study-exemplars

（该工具会调用它来构建对比语料库）配合使用。请先运行

preflight-check

——若存在直接被拒的缺陷，任何评分都毫无意义。

When to use

使用场景

The author wants a single, shareable readiness signal before submitting.
The author asks how their draft stacks up against the venue's strongest recent papers.
The author wants to know which dimensions are weakest relative to exemplars, ranked by fixability.

作者希望在提交前获得单一、可分享的准备情况信号。
作者想了解自己的草稿与该会议近期优秀论文相比表现如何。
作者想知道哪些维度与样本论文差距最大，并按可修复性排序。

When NOT to use it (say this plainly to the user)

不适用场景（需直白告知用户）

It cannot judge whether the science is novel, correct, or important — the things that actually win awards. Use
```
simulate-reviewers
```
for content critique.
It cannot predict acceptance or a best-paper award. Anyone who claims a tool can is selling false precision.
A high score on a flawed paper is meaningless. Conformance to form ≠ quality of substance.

无法判断研究内容是否新颖、正确或重要——这些才是真正决定能否获奖的因素。如需内容评审，请使用
```
simulate-reviewers
```
。
无法预测录用结果或最佳论文奖项。任何声称工具能做到这一点的说法都是在夸大其准确性。
存在缺陷的论文即使获得高分也毫无意义。形式上的一致性≠内容质量。

Inputs

输入项

The draft: a
```
.tex
```
file (or compiled PDF / markdown), with
```
.bib
```
if available.
The target venue id (e.g.
```
sigspatial-2026
```
) → its profile in
```
venues/
```
.
Optional: a corpus size N (default 8 exemplars) and a recency window (default last 5 years).

论文草稿：
```
.tex
```
文件（或编译后的PDF/Markdown），如有
```
.bib
```
文件也需提供。
目标会议ID（例如
```
sigspatial-2026
```
）→ 对应
```
venues/
```
目录中的配置文件。
可选参数：语料库规模N（默认8篇样本论文）和时间范围（默认最近5年）。

Process

流程

Resolve the venue profile from
```
venues/conferences/<venue>.yml
```
and its
```
family:
```
profile in
```
venues/families/<family>.yml
```
. If missing, ask the user or have
```
add-venue-profile
```
create it. Re-verify the venue is correct before scoring. Apply the staleness gate: profiles are year-versioned (
```
verified.valid_window
```
,
```
verified.last_verified_against_cfp
```
). Do not assert a hard format constraint (page limit, column count, mandatory section, deadline) from a profile whose
```
valid_window
```
does not include the target cycle without a fresh CFP check first; if you cannot check, mark it
```
needs-verification
```
and disclose that the basis was a year-mismatched profile (see references/scoring-rubric.md "Staleness gate"). A year-mismatched profile may still inform priors (the exemplar distribution, the modal skeleton).
Build the on-family exemplar corpus. The distribution dimensions (section architecture, citation density, abstract structure, figure/table conventions) must be scored against an on-family distribution — same venue family — never an off-family proxy. Resolve it in priority order: (a) a live
```
study-exemplars
```
corpus for the target venue — preferred; invoke
```
study-exemplars
```
to fetch (on demand, legally, transiently) N recent best-paper awardees and top-cited papers at the venue and extract their feature profile, never bundling or storing paper text; (b) the family profile's
```
exemplar_distribution
```
block as a fallback prior (disclose it is from the family prior, carry its confidence); (c) if neither exists for this family, reduce N and disclose
no on-family exemplar distribution
on those dimensions — do not borrow another family's numbers. If award lists are unavailable, fall back to top-cited and say so. The corpus basis (live vs family-prior vs none) must be disclosed.
Extract the draft's comparable features along the dimensions in references/scoring-rubric.md (section architecture, contribution framing, evaluation rigor signals, claim/citation density, abstract structure, figure/table conventions, reproducibility artifacts). Use
```
verify-citations
```
output if present so the citation-integrity dimension is grounded. Detect the realization level of each dimension (
```
absent
```
/
```
planned
```
/
```
drafted
```
/
```
complete
```
): a dimension whose evidence is dominated by honest
```
[RESULT]
```
/
```
[TBD]
```
/
```
\todo
```
placeholders is
```
planned
```
, not weak. Record it in the features JSON
```
realization
```
field.
Score each dimension as conformance to the exemplar distribution, with an explicit basis for every number (what was measured, what the exemplar range was). For
```
planned
```
/
```
absent
```
dimensions, score the completeness and specificity of the design (named baselines, datasets, ablation list, matched-budget protocol, variance/significance policy) — never realized numbers — and mark deferred
```
[RESULT]
```
slots as deferred-but-specified in the basis. Run
```
python3 scripts/scorecard.py features.json --venue <id>
```
to compute and render deterministically — do not eyeball the aggregate. The script caps planned dimensions below the "within range" band and relabels the index plan-conformance when any dimension is unrealized, so it is never silently compared against an executed paper.
Report: the scorecard (per-dimension 0–10 + venue-fit/plan-conformance index), the 3 weakest dimensions ranked by fixability, concrete fixes tied to exemplar patterns, and the mandatory caveat block. For a plan, frame the gap as experiments not run (path: specify → run), not design is weak. Offer the one-line shareable summary the user can screenshot.

解析会议配置文件：从
```
venues/conferences/<venue>.yml
```
及其
```
family:
```
字段对应的
```
venues/families/<family>.yml
```
配置文件中读取信息。若配置文件缺失，请询问用户或调用
```
add-venue-profile
```
创建。打分前需再次确认会议信息是否正确。启用过期检查机制：配置文件按年份版本化（
```
verified.valid_window
```
、
```
verified.last_verified_against_cfp
```
）。若配置文件的
```
valid_window
```
不包含目标投稿周期，且未重新核对最新征稿启事（CFP），则不得断言任何硬性格式约束（页数限制、栏数、必填章节、截止日期）；若无法核对，请标记为
```
needs-verification
```
并说明评分依据是年份不匹配的配置文件（详见references/scoring-rubric.md中的“过期检查机制”）。年份不匹配的配置文件仍可作为参考（样本论文分布、典型结构框架）。
构建同系列会议样本语料库：维度分布（章节架构、引用密度、摘要结构、图表规范）必须基于同系列会议的分布进行评分——不得使用跨系列会议的替代数据。按优先级顺序获取：(a) 目标会议的实时
```
study-exemplars
```
语料库——优先选择；调用
```
study-exemplars
```
按需、合法、临时获取该会议近期N篇最佳论文和高引用论文，并提取其特征配置，严禁打包或存储论文文本；(b) 同系列会议配置文件中的
```
exemplar_distribution
```
模块作为备用参考（需说明依据是同系列会议参考数据，并标注可信度）；(c) 若该系列会议无上述两种数据，减少N并说明该维度“无同系列会议样本分布数据”——不得借用其他系列会议的数据。若获奖论文列表不可用，则退而求其次使用高引用论文，并予以说明。必须公开语料库的来源依据（实时同系列语料库、同系列参考数据或无数据）。
提取草稿的可对比特征：依据references/scoring-rubric.md中的维度（章节架构、贡献框架、评估严谨性信号、论点/引用密度、摘要结构、图表规范、可复现性成果）提取特征。若存在
```
verify-citations
```
的输出结果，可用于支撑引用完整性维度的评分。检测每个维度的完成状态（
```
absent
```
/
```
planned
```
/
```
drafted
```
/
```
complete
```
）：若某维度的内容主要是
```
[RESULT]
```
/
```
[TBD]
```
/
```
\todo
```
这类占位符，则标记为
```
planned
```
（计划中），而非“薄弱”。将状态记录在特征JSON的
```
realization
```
字段中。
分维度评分：基于与样本分布的一致性进行评分，每个分数都需明确说明依据（测量内容、样本范围）。对于
```
planned
```
/
```
absent
```
维度，评分依据是设计的完整性和明确性（指定的基准模型、数据集、消融实验列表、匹配预算的方案、方差/显著性策略）——而非实际完成的数据，并在依据中标记延迟的
```
[RESULT]
```
插槽为“延迟但已明确”。运行
```
python3 scripts/scorecard.py features.json --venue <id>
```
进行确定性计算和生成——不得主观估算总分。脚本会将计划中维度的分数限制在“符合范围”以下，若存在未完成维度，则将指数重新标记为计划一致性指数，避免与已完成论文进行无意义对比。
输出报告：评分卡（分维度0–10分 + 会议适配度/计划一致性指数）、按可修复性排序的3个最薄弱维度、结合样本模式的具体修复建议，以及强制声明的免责条款。对于计划/大纲阶段的草稿，需将差距描述为“未开展的实验”（解决路径：明确方案→执行实验），而非“设计薄弱”。提供用户可截图分享的单行总结。

Output

输出结果

scorecard.md

containing:

Venue-fit index (0–10) with a one-line plain-English band (e.g. "structurally in line with recent {venue} winners; evaluation section is the gap"). For a plan/outline the index is labeled plan-conformance and must not be compared against executed papers.
Per-dimension table: score, realization level, exemplar range, your value, basis.
Top fixes ranked by impact × ease.
Corpus disclosure: which papers formed the basis, award-vs-cited, recency, and whether the distribution was a live on-family corpus, a family-profile prior, or absent (
```
no on-family exemplar distribution
```
).
Format-basis disclosure when any hard constraint came from a year-mismatched profile (flag it
```
needs-verification
```
until checked against the live CFP).
Caveats (always): conformance not quality; not an acceptance/award prediction; human judgment required.

生成

scorecard.md

文件，包含：

会议适配度指数（0–10分）及单行直白描述（例如“结构符合近期{venue}获奖论文的标准；评估章节是主要差距”）。对于计划/大纲阶段的草稿，指数标记为计划一致性指数，且不得与已完成论文对比。
分维度表格：分数、完成状态、样本范围、你的草稿数值、评分依据。
优先修复建议：按影响×易实现性排序。
语料库说明：构成语料库的论文来源、获奖论文vs高引用论文、时间范围，以及分布数据是来自实时同系列语料库、同系列参考数据还是无数据（
```
no on-family exemplar distribution
```
）。
格式依据说明：若任何硬性约束来自年份不匹配的配置文件，需标记为
```
needs-verification
```
，直至核对最新CFP。
免责声明（必须包含）：仅衡量一致性而非质量；不是对录用/获奖的预测；需结合人工判断。

Guardrails

约束规则

Every score states what was measured and the exemplar range it was compared against. No bare numbers.
Never present the index as a probability of acceptance or an award. Refuse to, if asked — explain why.
Corpus is fetched on demand from open-access sources, processed transiently, never stored or committed (see
```
study-exemplars
```
).
Never fabricate exemplar data to fill the corpus; if you can't reach N papers, score against fewer and disclose it.
This is a copilot signal, not a verdict. The author decides.

每个分数都需说明测量内容及对比的样本范围，不得仅给出数字。
绝不能将指数表述为录用或获奖的概率。若被要求这样做，需拒绝并解释原因。
语料库从开放获取来源按需获取，仅临时处理，绝不存储或提交（详见
```
study-exemplars
```
）。
不得编造样本数据来填充语料库；若无法获取N篇论文，则基于更少样本评分并予以说明。
这是辅助信号，而非最终结论。最终决定权在作者手中。

Memory

记忆机制

Uses the shared

.paper-memory/

convention in the user's paper directory (full spec:

paper-memory-convention.md

At start: read
```
.paper-memory/profile.yml
```
(vertical, venue tier) to pick exemplar emphasis, and
```
lessons.md
```
to recall which dimensions were weak last run and any
```
recurring
```
gaps for this author — lead with them.
At end: append durable findings in the shared format
```
- [YYYY-MM-DD] (benchmark-paper | <scope>) weak-dimension -> recommendation
```
(via
```
reflect-and-improve
```
's
```
reflect_log.py append
```
, which dedupes and dates). A dimension that lags exemplars across drafts is
```
recurring
```
; a one-time gap is
```
this-paper
```
. Do not log the full scorecard, only the lasting takeaways.
Create
```
.paper-memory/
```
on demand if absent and offer to add it to the project
```
.gitignore
```
. It is local-only; never upload it or copy it into this repo.

使用用户论文目录中的共享

.paper-memory/

约定（完整规范：

paper-memory-convention.md

）。

启动时：读取
```
.paper-memory/profile.yml
```
（研究方向、会议层级）以确定样本论文的侧重方向，读取
```
lessons.md
```
以回顾上次运行时的薄弱维度及该作者的
```
recurring
```
（反复出现的）差距——优先关注这些内容。
结束时：以共享格式将持久化结论追加记录：
```
- [YYYY-MM-DD] (benchmark-paper | <scope>) weak-dimension -> recommendation
```
（通过
```
reflect-and-improve
```
的
```
reflect_log.py append
```
实现，该工具会自动去重并添加日期）。若某维度在多篇草稿中均落后于样本论文，则标记为
```
recurring
```
；仅单篇草稿存在的差距标记为
```
this-paper
```
。不得记录完整评分卡，仅记录可复用的结论。
若
```
.paper-memory/
```
目录不存在，则按需创建，并建议用户将其添加到项目的
```
.gitignore
```
中。该目录仅本地存储；不得上传或复制到本仓库。