benchmark-and-mms-planner

Compare original and translation side by side

🇺🇸

Original

English

🇨🇳

Translation

Chinese

Benchmark And MMS Planner

基准测试与MMS规划器

Goal

目标

Design a verification and validation plan before trusting simulation results. The skill helps agents choose manufactured solutions, benchmark cases, refinement protocols, uncertainty checks, and pass/fail criteria.

在信任模拟结果之前设计验证与确认计划。该Skill可帮助Agent选择MMS（制造解法）、基准测试案例、细化协议、不确定性检查及通过/不通过准则。

Requirements

要求

Python 3.10+
No external dependencies
Works on Linux, macOS, and Windows

Python 3.10+
无外部依赖
支持Linux、macOS和Windows系统

Inputs to Gather

需要收集的输入

Input	Description	Example
PDE or model class	Governing family	`diffusion` , `elasticity` , `phase-field`
Quantity of interest	Metric to validate	`interface velocity` , `L2 temperature error`
Dimension	1, 2, or 3	`2`
Expected order	Formal discretization order	`2`
Reference availability	Analytic, benchmark, or none	`analytic`
Risk level	Cost or consequence of wrong result	`high`

输入项	描述	示例
PDE或模型类别	控制方程族	`diffusion` , `elasticity` , `phase-field`
关注量	待验证的指标	`interface velocity` , `L2 temperature error`
维度	1、2或3	`2`
预期阶数	形式离散阶数	`2`
参考依据可用性	解析解、基准测试或无	`analytic`
风险等级	结果错误的成本或影响	`high`

Decision Guidance

决策指南

Use MMS when code correctness is uncertain and an analytic solution can be injected.
Use canonical benchmarks when physical model validation matters more than code verification.
Use grid/time refinement whenever the result is used for a claim, design decision, or comparison.
Use uncertainty propagation when inputs are calibrated, noisy, or experimentally measured.

当代码正确性存疑且可注入解析解时，使用MMS。
当物理模型验证比代码验证更重要时，使用标准基准测试。
当结果用于声明、设计决策或对比时，务必使用网格/时间细化。
当输入经过校准、含噪声或为实验测量值时，使用不确定性传播。

Script Outputs

脚本输出

scripts/benchmark_mms_planner.py

emits

inputs

and

results

with:

```
verification_strategy
```
```
effective_model
```
— the resolved model family actually used; unknown families fall back to
```
general
```
.
```
mms_plan
```
```
benchmark_cases
```

refinement_protocol

(

dimension

levels

spacing_ratio

expected_order

accept_observed_order_min

include_time_refinement

)

uncertainty_plan

(

propagate_inputs

report_error_bars

separate_discretization_and_model_error

) — propagation/error-bar guidance driven by risk level and reference type.

```
acceptance_criteria
```
```
warnings
```

The

accept_observed_order_min

is an engineering screening heuristic, not a certified bound: it is the formal

expected_order

reduced by a fractional tolerance (10% for high risk, 20% otherwise) and floored at first-order convergence (

1.0

). The relative band keeps strictness consistent across formal orders. See

references/vv_patterns.md

scripts/benchmark_mms_planner.py

会输出

inputs

和

results

，包含以下内容：

```
verification_strategy
```
```
effective_model
```
— 实际使用的已解析模型族；未知模型族会 fallback 为
```
general
```
。
```
mms_plan
```
```
benchmark_cases
```

refinement_protocol

（

dimension

levels

spacing_ratio

expected_order

accept_observed_order_min

include_time_refinement

）

uncertainty_plan

（

propagate_inputs

report_error_bars

separate_discretization_and_model_error

）—— 由风险等级和参考类型驱动的传播/误差棒指南。

```
acceptance_criteria
```
```
warnings
```

accept_observed_order_min

是一种工程筛选启发式规则，而非认证边界：它是将形式

expected_order

减去一个分数容差（高风险为10%，其他为20%）后的值，且下限为一阶收敛（

1.0

）。相对容差带可确保不同形式阶数下的严格性一致。详见

references/vv_patterns.md

。

Workflow

工作流

Collect the governing model, quantity of interest, and risk level.
Run
```
benchmark_mms_planner.py --json
```
.
Treat warnings as blockers for high-risk claims.
Convert the returned protocol into tests, simulation runs, or review checklist items.

bash

python3 skills/verification-validation/benchmark-and-mms-planner/scripts/benchmark_mms_planner.py \
  --model diffusion \
  --quantity "L2 error in temperature" \
  --dimension 2 \
  --expected-order 2 \
  --reference analytic \
  --risk high \
  --json

收集控制模型、关注量及风险等级。
运行
```
benchmark_mms_planner.py --json
```
。
对于高风险声明，需将警告视为阻塞项。
将返回的协议转换为测试用例、模拟运行或评审检查项。

bash

python3 skills/verification-validation/benchmark-and-mms-planner/scripts/benchmark_mms_planner.py \
  --model diffusion \
  --quantity "L2 error in temperature" \
  --dimension 2 \
  --expected-order 2 \
  --reference analytic \
  --risk high \
  --json

Error Handling

错误处理

If the dimension or expected order is invalid, stop and correct the model description.
If no reference exists, use conservation and convergence checks but do not call the result validated.

若维度或预期阶数无效，需停止并修正模型描述。
若无参考依据，可使用守恒性和收敛性检查，但不得称结果已验证。

Limitations

局限性

This skill plans verification work; it does not run the solver or prove that a physical model is appropriate for an experiment.

该Skill仅规划验证工作；不运行求解器，也不证明物理模型适用于实验。

Verification checklist

验证检查清单

Before trusting a result that used this planner, record concrete evidence for each item:

Ran
```
benchmark_mms_planner.py --json
```
and saved the
```
inputs
```
block, confirming the echoed
```
dimension
```
,
```
expected_order
```
,
```
reference
```
, and
```
risk
```
match the actual run (a fallback to
```
effective_model: general
```
was intentional, not a typo in
```
--model
```
).
Executed the
```
refinement_protocol
```
: used the reported
```
levels
```
(3, or 4 for high risk) of systematically refined grids at
```
spacing_ratio
```
2, and recorded the observed order of accuracy from those runs.
Confirmed the observed order is at or above
```
accept_observed_order_min
```
; if below, logged the investigation (mesh not yet asymptotic, boundary/source errors, limiter activation) rather than treating the result as passed.
When
```
include_time_refinement
```
is
```
true
```
, ran a separate time-step refinement study and recorded the temporal observed order, not just the spatial one.
When
```
mms_plan.manufacture_solution
```
is
```
true
```
, derived the symbolic source/forcing term, applied the matching boundary terms, and recorded the L2 and Linf error norms versus the manufactured solution.
Checked every
```
acceptance_criteria
```
item with a number: conservation/balance closes within a documented tolerance, the quantity of interest plateaus under refinement, and any benchmark discrepancy from
```
benchmark_cases
```
is explained before production use.
Treated all entries in
```
warnings
```
as blockers for high-risk claims and recorded how each was resolved (e.g. an independent analytic/published reference was added when
```
reference
```
was
```
none
```
or
```
experimental
```
).

在信任使用本规划器得到的结果前，需为以下每一项记录具体证据：

运行
```
benchmark_mms_planner.py --json
```
并保存
```
inputs
```
块，确认回显的
```
dimension
```
、
```
expected_order
```
、
```
reference
```
和
```
risk
```
与实际运行匹配（若fallback到
```
effective_model: general
```
是有意为之，而非
```
--model
```
参数输入错误）。
执行
```
refinement_protocol
```
：使用报告的
```
levels
```
（高风险为4，其他为3）个系统细化网格，网格间距比为
```
spacing_ratio
```
=2，并记录这些运行得到的观测精度阶数。
确认观测阶数大于等于
```
accept_observed_order_min
```
；若小于，需记录调查结果（网格尚未进入渐近范围、边界/源项误差、限制器激活等），不得将结果视为通过。
当
```
include_time_refinement
```
为
```
true
```
时，需单独运行时间步长细化研究，并记录时间维度的观测阶数，而非仅空间维度。
当
```
mms_plan.manufacture_solution
```
为
```
true
```
时，推导符号源项/强迫项，应用匹配的边界条件，并记录与制造解法对比的L2和Linf误差范数。
检查所有带数值的
```
acceptance_criteria
```
项：守恒/平衡在文档化容差范围内闭合，关注量在细化后趋于平稳，且
```
benchmark_cases
```
中的任何基准测试差异在投入生产使用前已得到解释。
对于高风险声明，需将
```
warnings
```
中的所有条目视为阻塞项，并记录每项的解决方式（例如，当
```
reference
```
为
```
none
```
或
```
experimental
```
时，添加了独立的解析/已发表参考依据）。

Common pitfalls & rationalizations

常见误区与合理化借口

Tempting shortcut	Why it's wrong / what to do
"The planner ran and printed a plan, so the result is verified."	The script only plans V&V; it never runs the solver. Verification comes from executing the `refinement_protocol` , MMS, and `acceptance_criteria` , not from generating the plan.
"Two grids converged, so the observed order is fine."	`refinement_protocol.levels` is 3 (4 for high risk) for a reason: you need >=3 systematically refined grids to estimate observed order and confirm the solution is in the asymptotic range before quoting it.
"Observed order beats `accept_observed_order_min` , so it's certified."	That threshold is an engineering screening heuristic (formal order minus a 10%/20% relative tolerance, floored at 1.0), not a certified bound. For rigorous order verification run a Richardson/GCI study.
"Steady-looking model, so I can skip `include_time_refinement` ."	If the planner set `include_time_refinement: true` (any time-dependent or `general` fallback family), spatial refinement alone hides temporal error — run the time-step study too.
"We matched a benchmark, so the code is validated."	Matching `benchmark_cases` or converging shows the code approaches some solution; it does not prove the physical model is correct. Validation needs an independent reference plus model-error separation, not convergence alone.
" `reference none` is fine, the runs look physical."	With `reference: none` the strategy is verification-only; `warnings` says so. You may report convergence and conservation but must NOT call the result validated.
"Unknown model name, so I'll ignore the `general` fallback."	An unrecognized `--model` silently resolves to `effective_model: general` ; confirm that fallback is intended, since it changes both `benchmark_cases` and the time-refinement decision.

诱人的捷径	错误原因/正确做法
"规划器已运行并输出计划，所以结果已验证。"	脚本仅规划验证与确认（V&V）工作；从不运行求解器。验证来自执行 `refinement_protocol` 、MMS和 `acceptance_criteria` ，而非生成计划。
"两个网格已收敛，所以观测阶数没问题。"	`refinement_protocol.levels` 设为3（高风险为4）是有原因的：你需要至少3个系统细化的网格来估计观测阶数，并确认解决方案处于渐近范围后才能引用。
"观测阶数超过 `accept_observed_order_min` ，所以已认证。"	该阈值是一种工程筛选启发式规则（形式阶数减去10%/20%的相对容差，下限为1.0），而非认证边界。若需严格的阶数验证，请运行Richardson/GCI研究。
"模型看起来是稳态的，所以可以跳过 `include_time_refinement` 。"	如果规划器设置 `include_time_refinement: true` （任何时间相关或fallback为 `general` 的模型族），仅空间细化会隐藏时间误差——也需运行时间步长研究。
"我们匹配了基准测试，所以代码已验证。"	匹配 `benchmark_cases` 或收敛仅表明代码趋近于某个解；并不证明物理模型正确。验证需要独立参考依据加上模型误差分离，而非仅收敛。
" `reference none` 没问题，运行结果看起来符合物理规律。"	当 `reference: none` 时，策略仅为验证； `warnings` 中会说明这一点。你可以报告收敛性和守恒性，但不得称结果已验证。
"模型名称未知，所以我忽略 `general` fallback。"	未识别的 `--model` 会自动resolve为 `effective_model: general` ；需确认该fallback是有意为之，因为它会改变 `benchmark_cases` 和时间细化决策。

Security

安全性

Input Validation

输入验证

All inputs are command-line arguments parsed by

argparse

; validation happens in

plan_vv

(and partly in the parser). Any rejected input causes the script to print the error to stderr and exit with code

```
dimension
```
must be exactly
```
1
```
,
```
2
```
, or
```
3
```
; any other integer is rejected.
```
expected_order
```
must be a positive, finite number (NaN, infinity, zero, and negatives are rejected).
```
risk
```
must be one of the allowlist
```
low
```
,
```
medium
```
,
```
high
```
(enforced both as an
```
argparse
```
choice and re-checked in
```
plan_vv
```
).
```
reference
```
must be one of the allowlist
```
analytic
```
,
```
benchmark
```
,
```
experimental
```
,
```
none
```
(enforced both as an
```
argparse
```
choice and re-checked in
```
plan_vv
```
).
```
model
```
and
```
quantity
```
are capped at 256 characters (
```
MAX_FIELD_LEN
```
); longer strings are rejected.
The string content of
```
model
```
and
```
quantity
```
is otherwise not allowlisted or sanitized:
```
quantity
```
is echoed verbatim into the output, and an unrecognized
```
model
```
family is silently resolved to
```
general
```
rather than rejected.

所有输入均为

argparse

解析的命令行参数；验证在

plan_vv

中进行（部分在解析器中）。任何被拒绝的输入都会导致脚本将错误打印到stderr并以代码

退出。

```
dimension
```
必须严格为
```
1
```
、
```
2
```
或
```
3
```
；其他整数均会被拒绝。
```
expected_order
```
必须为正的有限数（NaN、无穷大、零和负数均会被拒绝）。
```
risk
```
必须为允许列表中的
```
low
```
、
```
medium
```
、
```
high
```
之一（同时作为
```
argparse
```
选项和在
```
plan_vv
```
中重新检查）。
```
reference
```
必须为允许列表中的
```
analytic
```
、
```
benchmark
```
、
```
experimental
```
、
```
none
```
之一（同时作为
```
argparse
```
选项和在
```
plan_vv
```
中重新检查）。
```
model
```
和
```
quantity
```
的长度上限为256字符（
```
MAX_FIELD_LEN
```
）；更长的字符串会被拒绝。
```
model
```
和
```
quantity
```
的字符串内容未做其他允许列表或 sanitization 处理：
```
quantity
```
会原样回显到输出中，未识别的
```
model
```
族会自动resolve为
```
general
```
而非被拒绝。

File Access

文件访问

The script reads and writes no files; all I/O is command-line args -> stdout JSON (or a short plain-text summary), with errors on stderr.
Because no filesystem paths are accepted or constructed, there is no path-traversal surface and no path-sandboxing logic is needed.
The only DoS-relevant size limit is the 256-character cap on
```
model
```
and
```
quantity
```
; numeric outputs are bounded by the validated inputs.

脚本不读取或写入任何文件；所有I/O均为命令行参数 -> stdout JSON（或简短的纯文本摘要），错误输出到stderr。
由于不接受或构造文件系统路径，因此不存在路径遍历风险，也无需路径沙箱逻辑。
唯一与DoS相关的大小限制是
```
model
```
和
```
quantity
```
的256字符上限；数值输出由已验证的输入限制。

Tool Restrictions

工具限制

The frontmatter declares

allowed-tools: Read, Bash, Write, Grep, Glob

```
Bash
```
is used solely to run the bundled
```
scripts/benchmark_mms_planner.py
```
(e.g. the
```
python3 ... --json
```
invocation in the Workflow).
```
Read
```
,
```
Grep
```
, and
```
Glob
```
are for inspecting the skill's own files and references (e.g.
```
references/vv_patterns.md
```
) when planning.
```
Write
```
supports turning the returned protocol into test stubs or checklist files; the planner script itself never writes.

前置声明

allowed-tools: Read, Bash, Write, Grep, Glob

。

```
Bash
```
仅用于运行捆绑的
```
scripts/benchmark_mms_planner.py
```
（例如工作流中的
```
python3 ... --json
```
调用）。
```
Read
```
、
```
Grep
```
和
```
Glob
```
用于在规划时检查Skill自身的文件和参考依据（例如
```
references/vv_patterns.md
```
）。
```
Write
```
支持将返回的协议转换为测试存根或检查清单文件；规划器脚本本身从不写入文件。

Safety Measures

安全措施

No
```
eval
```
,
```
exec
```
, or dynamic code execution; the planner is pure Python computing a dictionary.
The script spawns no subprocesses and invokes no external solvers, so there are no subprocess argument lists to escape.
No
```
pickle
```
or other deserialization of untrusted data is performed; output is serialized with
```
json.dumps
```
.
The 256-character field cap is the explicit DoS guard against pathological input strings.

无
```
eval
```
、
```
exec
```
或动态代码执行；规划器是纯Python实现，仅计算字典。
脚本不生成子进程，也不调用外部求解器，因此不存在子进程参数列表需要转义的问题。
不执行
```
pickle
```
或其他不可信数据的反序列化；输出使用
```
json.dumps
```
序列化。
256字符的字段上限是针对异常输入字符串的明确DoS防护措施。

References

参考依据

See
```
references/vv_patterns.md
```
for MMS, benchmark, and uncertainty planning notes.

有关MMS、基准测试和不确定性规划的说明，请参阅
```
references/vv_patterns.md
```
。

Version History

版本历史

1.1.3: Add a "Verification checklist" (evidence-based items tied to the planner's
```
refinement_protocol
```
,
```
mms_plan
```
,
```
acceptance_criteria
```
, and
```
warnings
```
) and a "Common pitfalls & rationalizations" table that pins down domain-specific V&V shortcuts (plan != verification, >=3 grids for observed order, screening band is not a certified bound, time refinement, convergence != validation,
```
reference none
```
,
```
general
```
fallback).
1.1.1: Make the eval suite discriminating by adding deterministic
```
script_checks
```
that pin the planner's specific output (resolved
```
verification_strategy
```
, the relative
```
accept_observed_order_min
```
band, refinement
```
levels
```
,
```
include_time_refinement
```
,
```
uncertainty_plan
```
flags, model-specific benchmark cases, and the exact warning strings) for each of the three cases.
1.1.0: Resolve unknown model families to
```
general
```
once so benchmark selection and the time-refinement decision agree (transient unlisted PDEs no longer skip time refinement); echo the resolved family as
```
effective_model
```
. Replace the fixed absolute observed-order offset with a relative tolerance floored at first order. Document
```
uncertainty_plan
```
,
```
effective_model
```
, and the acceptance heuristic. Add 256-character caps on string inputs.
1.0.0: Initial benchmark and MMS planning skill.

1.1.3：添加“验证检查清单”（与规划器的
```
refinement_protocol
```
、
```
mms_plan
```
、
```
acceptance_criteria
```
和
```
warnings
```
绑定的循证项）和“常见误区与合理化借口”表格，明确领域特定的V&V捷径（计划≠验证、至少3个网格用于观测阶数、筛选带不是认证边界、时间细化、收敛≠验证、
```
reference none
```
、
```
general
```
fallback）。
1.1.1：通过添加确定性
```
script_checks
```
使评估套件更具判别力，针对三种案例分别固定规划器的特定输出（已解析的
```
verification_strategy
```
、相对
```
accept_observed_order_min
```
容差带、细化
```
levels
```
、
```
include_time_refinement
```
、
```
uncertainty_plan
```
标志、模型特定的基准测试案例及确切的警告字符串）。
1.1.0：将未知模型族一次性resolve为
```
general
```
，使基准测试选择和时间细化决策保持一致（未列出的瞬态PDE不再跳过时间细化）；将已解析的族回显为
```
effective_model
```
。将固定的绝对观测阶数偏移替换为下限为一阶的相对容差。记录
```
uncertainty_plan
```
、
```
effective_model
```
和验收启发式规则。为字符串输入添加256字符上限。
1.0.0：初始版本的基准测试与MMS规划Skill。