Back to Details

mllm-eval

Compare original and translation side by side

🇺🇸

Original

English
🇨🇳

Translation

Chinese

MLLM-Eval Skill

MLLM-Eval 技能

Purpose

用途

This skill makes an LLM / MLLM clinical evaluation defensible: a real adjudicated reference standard, faithfulness measured not assumed, clinical-efficacy metrics beyond n-gram overlap, a pretraining- contamination check, prompt-sensitivity disclosed, and a reader study where text is generated. It is model-agnostic — every check applies to a closed API and to open weights — and read-only (an advisory design/audit skill): it audits the evaluation design and specifies and routes the clinical-efficacy metrics (RadGraph-F1 / CheXbert-F1 via their published extractors) rather than running the model or computing the metrics itself.
It is the LLM/MLLM evaluation-design counterpart in the lane — an auditor that hands the specified metrics to their extractors and 
/analyze-stats
, parallel to how 
/model-validation
audits an imaging model's design (the imaging metrics themselves are computed by 
/model-evaluation
). The reviewer-side audit of a finished manuscript uses the 
mllm_evaluation.md
(ME0–ME8) probe via 
/self-review
and 
/peer-review
; this skill is the author-side harness design. It routes the reader study to 
/design-ai-benchmarking
, the sizing to 
/calc-sample-size
, and TRIPOD-LLM / MI-CLEAR-LLM compliance to 
/check-reporting
.
本技能让LLM / MLLM临床评估具备可辩护性：采用真实的裁决参考标准，真实性可测量而非假设，使用超越n元语法重叠的临床疗效指标，进行预训练污染检查，披露提示敏感性，以及针对生成文本开展读者研究。它具有模型无关性——所有检查适用于闭源API和开源权重模型——且为只读型（一种咨询式设计/审核技能）：它会审核评估设计，并指定和对接临床疗效指标（通过其公开的提取器获取RadGraph-F1 / CheXbert-F1），而非运行模型或自行计算指标。
它是该领域内LLM/MLLM的评估设计对应工具——作为审核方将指定的指标交付给对应的提取器和
/analyze-stats
，与
/model-validation
审核影像模型设计的方式类似（影像指标本身由
/model-evaluation
计算）。针对已完成稿件的审稿方审核，会通过
/self-review
和
/peer-review
使用
mllm_evaluation.md
（ME0–ME8）探针；本技能则是作者端的工具框架设计。它会将读者研究对接至
/design-ai-benchmarking
，样本量计算对接至
/calc-sample-size
，TRIPOD-LLM / MI-CLEAR-LLM合规性检查对接至
/check-reporting
。

When to use

适用场景

  • You are designing or auditing an evaluation of an LLM/MLLM on a clinical task and want it to cover the axes a reviewer will check (reference standard, faithfulness, contamination, prompt sensitivity, reader study).
  • 你正在设计或审核LLM/MLLM在临床任务上的评估，希望覆盖审稿人会检查的维度（参考标准、真实性、污染情况、提示敏感性、读者研究）。

When NOT to use

不适用场景

  • AI-vs-human-expert benchmark with a rated rubric → 
    /design-ai-benchmarking
    .
  • Imaging prediction/segmentation model → 
    /model-evaluation
    + 
    /model-validation
    .
  • Image-to-image generative model → the 
    image_synthesis
    probe.
  • Training / serving the LLM → out of scope.
  • Item-level TRIPOD-LLM / MI-CLEAR-LLM audit of a finished manuscript → 
    /check-reporting
    .
  • 采用评分 rubric 的AI vs 人类专家基准测试 → 请使用
    /design-ai-benchmarking
    。
  • 影像预测/分割模型 → 请使用
    /model-evaluation
    + 
    /model-validation
    。
  • 图像到图像生成模型 → 请使用
    image_synthesis
    探针。
  • LLM的训练/部署 → 超出本技能范围。
  • 针对已完成稿件的条目级TRIPOD-LLM / MI-CLEAR-LLM审核 → 请使用
    /check-reporting
    。

Workflow

工作流程

Phase 1 — Pin the task, model, comparator, decoding (ME0)

阶段1 — 明确任务、模型、对比项、解码方式（ME0）

State the task (report generation / VQA / extraction-classification), the exact model + version/date (closed API or open-weights id), the decoding settings (temperature, seed, max tokens), and what the outputs are scored against.
说明任务类型（报告生成 / VQA / 提取-分类）、具体模型+版本/日期（闭源API或开源权重ID）、解码设置（温度、随机种子、最大token数），以及输出结果的评分依据。

Phase 2 — Reference standard + metrics (ME1, ME2)

阶段2 — 参考标准 + 指标（ME1, ME2）

Require an adjudicated expert reference (not a single unverified report or a model-derived label). For report generation, report a clinical-efficacy metric — RadGraph-F1 (Jain et al., NeurIPS 2021) or CheXbert-F1 (Smit et al., 2020), or the composite RadCliQ (Yu et al., Patterns 2023) — alongside any BLEU/ROUGE, with CIs. For VQA/classification, report accuracy at the real prevalence with a stated answer-matching rule.
要求采用经裁决的专家参考标准（而非单一未验证报告或模型生成的标签）。对于报告生成任务，需报告临床疗效指标——RadGraph-F1（Jain等人，NeurIPS 2021）或CheXbert-F1（Smit等人，2020），或复合指标RadCliQ（Yu等人，Patterns 2023）——同时报告BLEU/ROUGE等指标，并提供置信区间（CIs）。对于VQA/分类任务，需报告基于真实患病率的准确率，并说明答案匹配规则。

Phase 3 — Faithfulness + contamination (ME3, ME4)

阶段3 — 真实性 + 污染检查（ME3, ME4）

Add an atomic-fact faithfulness measure + a false-premise / abstention probe (MedVH, Med-HALT) — report a hallucination rate, not just accuracy. For any public benchmark (VQA-RAD, SLAKE, MIMIC-CXR- derived, MedQA), add a contamination statement: training cutoff vs benchmark release, a held-out / post-cutoff set, or a contamination probe.
添加原子事实真实性测量方法 + 错误前提/弃权探针（MedVH, Med-HALT）——报告幻觉率，而非仅报告准确率。对于任何公开基准（VQA-RAD, SLAKE, MIMIC-CXR衍生数据集, MedQA），需添加污染声明：训练截止日期与基准发布日期对比、预留/截止日期后数据集，或污染检测探针。

Phase 4 — Prompt sensitivity + reader study (ME5, ME7)

阶段4 — 提示敏感性 + 读者研究（ME5, ME7）

Disclose the exact prompt(s), temperature/seed, ≥ 3 runs with variance, and a prompt-robustness check. For a deployment/utility claim, design a blinded reader study with an error taxonomy (route the rubric/IRR to 
/design-ai-benchmarking
, ICC/κ to 
/analyze-stats
, sizing to 
/calc-sample-size
).
披露完整提示内容、温度/随机种子、≥3次运行的方差情况，以及提示鲁棒性检查。对于部署/实用性声明，需设计盲法读者研究并制定错误分类体系（将评分标准/信度检验对接至
/design-ai-benchmarking
，ICC/κ对接至
/analyze-stats
，样本量计算对接至
/calc-sample-size
）。

Phase 5 — Gate the plan (deterministic)

阶段5 — 方案把关（确定性）

bash
python3 ${CLAUDE_SKILL_DIR}/scripts/check_mllm_eval_completeness.py \
  --plan plan.md --task report_generation|vqa|classification --strict
NGRAM_ONLY
/ 
FAITHFULNESS_MISSING
/ 
REFERENCE_STANDARD_MISSING
/ 
CONTAMINATION_UNADDRESSED
/ 
READER_STUDY_MISSING
must be resolved.
bash
python3 ${CLAUDE_SKILL_DIR}/scripts/check_mllm_eval_completeness.py \
  --plan plan.md --task report_generation|vqa|classification --strict
必须解决
NGRAM_ONLY
/ 
FAITHFULNESS_MISSING
/ 
REFERENCE_STANDARD_MISSING
/ 
CONTAMINATION_UNADDRESSED
/ 
READER_STUDY_MISSING
这些问题。

Phase 6 — Hand off

阶段6 — 任务移交

Methods/Results → 
/write-paper
; compliance (TRIPOD-LLM / MI-CLEAR-LLM) → 
/check-reporting
; reviewer audit → 
/self-review
(loads ME0–ME8).
方法/结果部分 → 对接
/write-paper
；合规性检查（TRIPOD-LLM / MI-CLEAR-LLM）→ 对接
/check-reporting
；审稿人审核 → 对接
/self-review
（加载ME0–ME8）。

Anti-Hallucination

防幻觉规则

  • Never fabricate model outputs, reference labels, or metric scores. Compute only what the supplied outputs allow; if a reference standard or outputs are missing, say so and stop.
  • Never report n-gram overlap (BLEU/ROUGE) as clinical correctness — pair it with a clinical-efficacy metric, and flag the n-gram score for what it is.
  • Never claim "no contamination" without a stated check when a public benchmark is used.
  • If a metric (RadGraph-F1 / CheXbert-F1) or its extractor is uncertain, flag 
    [VERIFY]
    and ask rather than inventing a number.
  • 绝不编造模型输出、参考标签或指标分数。仅根据提供的输出进行计算；若缺少参考标准或输出数据，需说明并停止操作。
  • 绝不将n元语法重叠（BLEU/ROUGE）作为临床正确性的判定依据——需将其与临床疗效指标配对报告，并明确标注n元语法分数的实际意义。
  • 使用公开基准时，未进行明确检查绝不能声称“无污染”。
  • 若对某个指标（RadGraph-F1 / CheXbert-F1）或其提取器存在疑问，需标记
    [VERIFY]
    并询问，而非自行编造数值。

Deterministic gate

确定性把关工具

scripts/check_mllm_eval_completeness.py
— task-aware presence gate on the evaluation plan (stdlib, network-free). Reproducible challenge: 
bash ${CLAUDE_SKILL_DIR}/scripts/mllm_eval_completeness_challenge/verify.sh
.
scripts/check_mllm_eval_completeness.py
— 针对评估方案的任务感知型存在性检查工具（标准库，无需网络）。可复现性验证挑战： 
bash ${CLAUDE_SKILL_DIR}/scripts/mllm_eval_completeness_challenge/verify.sh
。

Boundaries

边界范围

mllm-eval (this skill: harness design + completeness gate, model-agnostic)
  ├─ design-ai-benchmarking (reader-study rubric / IRR)
  ├─ calc-sample-size (reader + case sizing)
  ├─ write-paper + check-reporting (TRIPOD-LLM / MI-CLEAR-LLM)
  └─ self-review / peer-review (ME0–ME8 reviewer probe)
mllm-eval（本技能：工具框架设计 + 完整性把关，模型无关）
  ├─ design-ai-benchmarking（读者研究评分标准 / 信度检验）
  ├─ calc-sample-size（读者与病例样本量计算）
  ├─ write-paper + check-reporting（TRIPOD-LLM / MI-CLEAR-LLM合规性）
  └─ self-review / peer-review（ME0–ME8审稿探针）

Reference Files

参考文件

  • ${CLAUDE_SKILL_DIR}/references/evaluation_axes.md
    — the why behind the ME2–ME7 axes: clinical-efficacy metrics beyond n-gram overlap (e.g. RadGraph-F1 / CheXbert-F1 vs BLEU/ROUGE), faithfulness & hallucination, pretraining/benchmark contamination, prompt-sensitivity & determinism, answer-matching, and the reader study — each mapped to its gate verdict. Load on demand during Phases 2–4.
  • ${CLAUDE_SKILL_DIR}/references/evaluation_axes.md
    — ME2–ME7各维度的设计依据：超越n元语法重叠的临床疗效指标（例如RadGraph-F1 / CheXbert-F1 vs BLEU/ROUGE）、真实性与幻觉、预训练/基准污染、提示敏感性与确定性、答案匹配以及读者研究——每个维度均对应其把关判定结果。可在阶段2–4按需加载。