model-evaluation

Compare original and translation side by side

🇺🇸

Original

English

🇨🇳

Translation

Chinese

Model-Evaluation Skill

Model-Evaluation 技能

Purpose

用途

This skill makes a medical-imaging model's held-out evaluation task-correct and honest: the right metric for the task and the prevalence, with uncertainty, calibration, and subgroup performance. It emits a per-case metric table that the publication statistics build on, and gates the metric choice against Metrics Reloaded (Maier-Hein & Reinke et al., Nat Methods 2024) and CLAIM 2024.

It sits between

/model-validation

(which audits the split / design) and

/analyze-stats

(which owns the comparative inference). It computes the imaging-specific per-case metrics (surface distances, FROC, ECE of a softmax head);

/analyze-stats

owns DeLong / NRI / IDI / decision curves / MRMC. Like

/analyze-stats

, it generates and executes code on your predictions — numbers are never hand-typed.

该技能让医学影像模型的留出集评估符合任务要求且真实可靠：针对任务和患病率选择适配的指标，包含不确定性分析、校准和亚组性能数据。它输出一份单病例指标表，作为学术发表统计分析的基础，并依据Metrics Reloaded（Maier-Hein & Reinke等人，《Nature Methods》2024）和CLAIM 2024筛选指标。

该技能位于

/model-validation

（负责审核数据集划分/实验设计）与

/analyze-stats

（负责对比推断分析）之间。它计算影像领域专属的单病例指标（表面距离、FROC、softmax头的ECE）；

/analyze-stats

则负责DeLong检验、NRI、IDI、决策曲线、MRMC分析。与

/analyze-stats

类似，它会生成并执行代码处理你的预测结果——所有数值绝不手动输入。

When to use

适用场景

You have held-out predictions + ground truth and need task-correct metrics with CIs, calibration, and subgroup slices, plus a per-case table for the manuscript statistics.

你已有留出集预测结果与真实标签，需要符合任务要求的带置信区间的指标、校准数据、亚组分析结果，以及用于手稿统计的单病例表格。

When NOT to use

不适用场景

Auditing the validation design / leakage →
```
/model-validation
```
.
DeLong / NRI / IDI / decision curves / MRMC reader study →
```
/analyze-stats
```
.
Building / training the model →
```
/model-scaffold
```
; LLM / MLLM →
```
/mllm-eval
```
.
Figure rendering →
```
/make-figures
```
.

审核验证设计/数据泄露 → 使用
```
/model-validation
```
。
DeLong检验、NRI、IDI、决策曲线、MRMC阅片研究 → 使用
```
/analyze-stats
```
。
模型构建/训练 → 使用
```
/model-scaffold
```
；大语言模型/多模态大语言模型 → 使用
```
/mllm-eval
```
。
图表绘制 → 使用
```
/make-figures
```
。

Workflow

工作流程

Phase 1 — Fix the analysis unit and the task

阶段1——确定分析单元与任务

State the task (segmentation / classification / detection / interactive / generative) and the analysis unit the metric must respect (per-patient vs per-lesion vs per-image). A per-lesion metric must not be reported as per-patient.

明确任务类型（分割/分类/检测/交互式/生成式）以及指标必须遵循的分析单元（按患者/按病灶/按图像）。按病灶计算的指标不得按患者维度报告。

Phase 2 — Compute task-correct metrics

阶段2——计算符合任务要求的指标

Generate evaluation code that computes, on the held-out predictions:

segmentation: Dice/IoU and a boundary metric (HD95 / NSD), per structure not only a global mean, with bootstrap 95% CIs.
classification: AUROC and AUPRC with bootstrap CIs, sensitivity/specificity, and PPV/NPV at the deployment prevalence (not a balanced set).
detection: FROC / mAP with the IoU match criterion stated.
interactive / promptable segmentation (SAM2 / MedSAM2 / nnInteractive): the segmentation metrics above plus the interaction axis — Dice-vs-interactions / number-of-clicks (NoC) to a target threshold, initial-vs-converged (or peak) Dice, and per-case interaction/inference time (see the metric guide; the study design is in
```
/design-study
```
+
```
/model-validation
```
).
generative / synthesis (image generation or modification): full-reference similarity (MSE/RMSE/PSNR/SSIM) or no-reference quality (SNR/CNR, standardized visual scores), plus a downstream-task evaluation — image quality is not clinical utility (Park et al., Radiol Med 2024). For multiclass classification, state the aggregation scheme (one-vs-rest / macro / micro / pairwise / Obuchowski); time-to-event discrimination (Harrell's C, time-dependent ROC) is handed to
```
/analyze-stats
```
. Add calibration (reliability diagram / ECE) and subgroup slices (the Model Card Factors). See
```
${CLAUDE_SKILL_DIR}/references/metric_guide.md
```
. Emit a per-case CSV for
```
/analyze-stats
```
.

生成评估代码，对留出集预测结果计算以下指标：

分割任务：Dice/IoU 加上边界指标（HD95/NSD），按结构单独计算而非仅全局均值，附带bootstrap 95%置信区间。
分类任务：AUROC和AUPRC（带bootstrap置信区间）、灵敏度/特异性，以及部署患病率下的阳性预测值/阴性预测值（而非基于平衡数据集）。
检测任务：FROC/mAP，需明确IoU匹配标准。
交互式/可提示分割任务（SAM2/MedSAM2/nnInteractive）：上述分割指标加上交互维度数据——Dice与交互次数的关系、达到目标阈值所需点击次数（NoC）、初始与收敛（或峰值）Dice对比，以及单病例交互/推理耗时（详见指标指南；研究设计需结合
```
/design-study
```
+
```
/model-validation
```
）。
生成式/图像合成任务（图像生成或修改）：全参考相似度指标（MSE/RMSE/PSNR/SSIM）或无参考质量指标（SNR/CNR、标准化视觉评分），加上下游任务评估——图像质量不等于临床效用（Park等人，《Radiol Med》2024）。对于多分类任务，需明确聚合方案（一对一/宏平均/微平均/成对比较/Obuchowski法）；时间-事件区分度分析（Harrell's C、时间依赖ROC）需移交
```
/analyze-stats
```
处理。添加校准（可靠性图表/ECE）和亚组分析（模型卡片要素）。详见
```
${CLAUDE_SKILL_DIR}/references/metric_guide.md
```
。输出供
```
/analyze-stats
```
使用的单病例CSV文件。

Phase 3 — Gate the metric choice (deterministic)

阶段3——指标选择校验（确定性）

bash

python3 ${CLAUDE_SKILL_DIR}/scripts/check_metric_reporting.py \
  --report results.md --task segmentation|classification|detection|interactive|generative --strict

PIXEL_ACCURACY_SEG

NO_BOUNDARY_METRIC

ACCURACY_ONLY

DETECTION_METRIC_MISSING

must be zero.

bash

python3 ${CLAUDE_SKILL_DIR}/scripts/check_metric_reporting.py \
  --report results.md --task segmentation|classification|detection|interactive|generative --strict

PIXEL_ACCURACY_SEG

NO_BOUNDARY_METRIC

ACCURACY_ONLY

DETECTION_METRIC_MISSING

的校验结果必须为0。

Phase 4 — Hand off

阶段4——移交结果

The per-case table →

/analyze-stats

(DeLong / NRI / IDI / decision curves, publication tables); figures →

/make-figures

; the numbers + subgroup performance →

/model-card

; Methods/Results →

/write-paper

; compliance →

/check-reporting

单病例表格 →

/analyze-stats

（用于DeLong检验、NRI、IDI、决策曲线、发表用表格）；图表 →

/make-figures

；数值+亚组性能 →

/model-card

；方法/结果部分 →

/write-paper

；合规性检查 →

/check-reporting

。

Anti-Hallucination

防幻觉机制

Never fabricate a metric value. Every number comes from executed code on the supplied predictions; if predictions or ground truth are missing, say so and stop — do not invent a result.
Never report pixel/voxel accuracy for segmentation or bare accuracy under imbalance — the gate flags these; report Dice + a boundary metric, or AUROC + AUPRC with CIs.
Never report a per-lesion metric as if it were per-patient — respect the analysis unit.
If a metric definition or its CI method is uncertain, flag
```
[VERIFY]
```
and ask.

绝不编造指标数值。所有数值均来自对提供的预测结果执行代码后的输出；若缺少预测结果或真实标签，需说明并终止流程——不得虚构结果。
分割任务绝不报告像素/体素准确率，不平衡数据下绝不报告单纯准确率——校验工具会标记此类问题；应报告Dice+边界指标，或带置信区间的AUROC+AUPRC。
绝不将按病灶计算的指标按患者维度报告——需严格遵循分析单元要求。
若对指标定义或置信区间计算方法存疑，需标记
```
[VERIFY]
```
并询问。

Deterministic gate

确定性校验工具

scripts/check_metric_reporting.py

— flags a task-metric mismatch / missing uncertainty (stdlib, network-free). Reproducible challenge:

bash ${CLAUDE_SKILL_DIR}/scripts/metric_reporting_challenge/verify.sh

scripts/check_metric_reporting.py

——标记任务与指标不匹配/缺失不确定性分析的情况（基于标准库，无需网络）。可复现验证挑战：

bash ${CLAUDE_SKILL_DIR}/scripts/metric_reporting_challenge/verify.sh

。

Reference Files

参考文件

Load on demand (keep SKILL.md short):

```
${CLAUDE_SKILL_DIR}/references/metric_guide.md
```
— operational checklist: the task-correct metric per task (segmentation Dice + HD95/NSD per structure; classification AUROC + AUPRC + sens/spec at deployment prevalence; detection FROC/mAP with a stated IoU), plus calibration, subgroup slices, run-variance, and the per-case CSV hand-off.
```
${CLAUDE_SKILL_DIR}/references/metric_selection_grounding.md
```
— the standards grounding behind those choices: the Metrics Reloaded task-fingerprint principle, why each metric pairing is required, calibration vs discrimination, disaggregated reporting, and the CLAIM 2024 reporting-fit map (
```
/check-reporting
```
owns the item audit).

按需加载（保持SKILL.md简洁）：

```
${CLAUDE_SKILL_DIR}/references/metric_guide.md
```
——操作清单：各任务对应的合规指标（分割任务为Dice+HD95/NSD按结构计算；分类任务为AUROC+AUPRC+部署患病率下的灵敏度/特异性；检测任务为带明确IoU的FROC/mAP），加上校准、亚组分析、运行方差，以及供移交的单病例CSV文件规范。
```
${CLAUDE_SKILL_DIR}/references/metric_selection_grounding.md
```
——指标选择的标准依据：Metrics Reloaded的任务指纹原则、为何要求特定指标组合、校准与区分度的关系、分类报告，以及CLAIM 2024的报告适配映射（
```
/check-reporting
```
负责逐项审核）。

Boundaries

边界划分

model-validation (design) -> model-evaluation (this skill: per-case task-correct metrics + CIs)
  -> analyze-stats (DeLong / NRI / IDI / decision curves, publication tables) -> make-figures
  -> model-card (numbers + subgroup) -> write-paper + check-reporting

model-validation (设计) -> model-evaluation (本技能：单病例合规指标+置信区间)
  -> analyze-stats (DeLong检验/NRI/IDI/决策曲线、发表用表格) -> make-figures
  -> model-card (数值+亚组数据) -> write-paper + check-reporting