Back to Details

leaderboard

Compare original and translation side by side

🇺🇸

Original

English
🇨🇳

Translation

Chinese

nurb leaderboard

nurb 排行榜

Two generated surfaces, one editorial layer. 
evals/REPORT.md
(audit tables) and 
site/benchmarks.html
(the page behind nurb.dev/benchmarks) both regenerate mechanically from 
evals/submissions/
; the verdict sentences and subscription labels live in 
evals/src/nurb_evals/site.py
and are written by a person. Submission PRs are pure additions and merge freely; nothing reaches the public page until this skill runs. That gap is deliberate: it is where the sanity check and the verdicts happen, so publishing is an editorial act, not a side effect of merging.
两个生成的表面，一个编辑层。
evals/REPORT.md
（审核表）和
site/benchmarks.html
（nurb.dev/benchmarks对应的页面）均从
evals/submissions/
自动生成；评判语句和订阅标签存储在
evals/src/nurb_evals/site.py
中，由人工编写。提交PR仅为新增内容，可自由合并；在运行此技能之前，任何内容都不会发布到公开页面。这种间隔是有意设计的：健全性检查和评判结果在此阶段完成，因此发布是一项编辑行为，而非合并操作的副作用。

Step 1: Find what is new

步骤1：查找新增内容

The runs added since the page was last regenerated:
bash
git log -1 --format=%H -- site/benchmarks.html
git diff --stat <that-commit>..HEAD -- evals/submissions/
Each new directory under 
evals/submissions/
is one run: 
<harness>-<model>-<effort>-<hex>/
holding 
results.jsonl
plus per-trial transcripts and part sources. If nothing is new, say so and stop.
自页面上次生成以来新增的运行：
bash
git log -1 --format=%H -- site/benchmarks.html
git diff --stat <that-commit>..HEAD -- evals/submissions/
evals/submissions/
下的每个新目录对应一次运行：格式为
<harness>-<model>-<effort>-<hex>/
，包含
results.jsonl
以及每个测试的记录和部件源文件。如果没有新增内容，说明情况并停止操作。

Step 2: Sanity-check every new run

步骤2：对所有新增运行进行健全性检查

Work from 
evals/
(its own uv project; 
uv sync --locked
first if the venv is stale). For each new run directory:
  • Rows parse and carry full identity: every line of 
    results.jsonl
    has harness, harness_version, model, effort, seed, nurb_version, benchmark_version, a 12-char benchmark_revision, and timeout_s. A benchmark_revision that matches no revision this repo ever shipped is disqualifying.
  • Artifacts are complete and sanitized: a transcript and the part source for every row; no 
    /Users/
    , 
    /home/
    , or usernames anywhere (the suite's sanitization test enforces this too).
  • The parts are authored, not planted: hash the submitted part files against 
    evals/tests/solutions/
    and against parts from other submissions. An exact match with a reference solution is disqualifying; matches across unrelated submissions are worth reading.
  • Spot-check by re-grading: for at least one row per new run (and every row that looks too good), rebuild the trial project (
    task.materialize
    , drop the part in, restore the submitted 
    measurements.toml
    for leg_cup) and run the grader. The committed score must reproduce exactly; grading is deterministic.
  • Read one transcript per new contributor: the headless preamble held (no 
    nurb dev
    ), the model actually iterated, and the session matches the row's timings.
A run that fails a check is removed from the PR with a comment saying which check and why, not silently. Suspicion is not proof: when a re-grade mismatches, check the benchmark_revision first; a row graded under an older shipped revision reproduces under that revision's scorer, not today's.
在
evals/
目录下操作（这是一个独立的uv项目；如果虚拟环境已过期，先执行
uv sync --locked
）。针对每个新的运行目录：
  • 行数据可解析且包含完整标识：
    results.jsonl
    的每一行都包含harness、harness_version、model、effort、seed、nurb_version、benchmark_version、12位字符的benchmark_revision和timeout_s。如果benchmark_revision与仓库中已发布的任何版本都不匹配，则此运行不合格。
  • 工件完整且已清理：每一行都对应一份记录和部件源文件；内容中不得包含
    /Users/
    、
    /home/
    或用户名（测试套件的清理测试也会强制执行此要求）。
  • 部件为原创，非植入内容：将提交的部件文件与
    evals/tests/solutions/
    以及其他提交的部件进行哈希比对。如果与参考解决方案完全匹配，则不合格；不同提交之间的匹配情况需要进一步核查。
  • 通过重新评分进行抽样检查：每个新运行至少抽取一行（以及所有看起来异常优秀的行），重建测试项目（执行
    task.materialize
    ，放入部件，恢复提交的
    measurements.toml
    用于leg_cup）并运行评分器。提交的分数必须完全复现；评分是确定性的。
  • 阅读每位新贡献者的一份记录：确认无头预处理正常（未使用
    nurb dev
    ）、模型确实进行了迭代，且会话时间与行数据中的时间一致。
未通过检查的运行将从PR中移除，并附上说明具体检查项和失败原因的注释，不得静默处理。怀疑不等于证据：当重新评分结果不匹配时，首先检查benchmark_revision；基于旧版本评分的行数据，需使用对应版本的评分器复现，而非当前版本。

Step 3: Verdicts

步骤3：评判结果

Every (harness, model, effort) combo on the board should have an entry in 
VERDICTS
in 
evals/src/nurb_evals/site.py
: which subscription it runs on, and one or two sentences a person with a printer can act on. Ground every claim in the rows and transcripts (findings, timings, the honesty tasks); never speculate. A new combo without a verdict renders numbers-only, which is acceptable for a day, not a policy. Capped or censored data is named as such ("hit the session limit"), never averaged into a claim.
排行榜上的每个（harness、model、effort）组合都应在
evals/src/nurb_evals/site.py
的
VERDICTS
中有对应的条目：包括其运行所需的订阅，以及一两句可供用户参考的语句。所有表述都必须基于行数据和记录（发现结果、时间、诚信任务）；不得进行推测。新组合若无评判结果，将仅显示数值，这种情况可允许存在一天，但不能成为常规做法。对于受限或审查的数据需明确标注（如“达到会话限制”），不得将其纳入平均计算。

Step 4: Regenerate, look, publish

步骤4：重新生成、检查、发布

bash
cd evals
uv run python -m nurb_evals.report --write
uv run python -m nurb_evals.site
Open 
site/benchmarks.html
in a browser and look at it before publishing: label collisions on the chart, a card wrapping badly, an empty state showing when rows exist. Screenshot, not DOM-query. Then a PR (never straight to main): the regenerated pair, any verdict edits, and a body that lists which runs were published and which were rejected with reasons. 
uv run pytest -q
from 
evals/
must be green.
The page deploys with 
site/
however the site deploys; this skill's job ends at the merged PR.
bash
cd evals
uv run python -m nurb_evals.report --write
uv run python -m nurb_evals.site
发布前在浏览器中打开
site/benchmarks.html
进行检查：图表上的标签冲突、卡片排版异常、已有行数据但显示空状态等问题。需截图检查，而非仅通过DOM查询。然后创建PR（绝不直接提交到main分支）：包含重新生成的文件、任何评判结果的编辑内容，以及说明已发布运行和被拒绝运行及原因的PR正文。在
evals/
目录下执行
uv run pytest -q
必须全部通过。
页面将通过
site/
目录的部署流程进行部署；此技能的工作在PR合并后结束。