hpc-runtime-doctor

Compare original and translation side by side

🇺🇸

Original

English

🇨🇳

Translation

Chinese

HPC Runtime Doctor

HPC运行时诊断器

Goal

目标

Turn cluster symptoms into a resource-layout diagnosis, environment checklist, and safe retry plan.

将集群作业的异常症状转化为资源布局诊断结果、环境检查清单及安全重试方案。

Requirements

要求

Python 3.10+
No external dependencies
Works on Linux, macOS, and Windows

Python 3.10+
无外部依赖
支持Linux、macOS和Windows系统

Inputs to Gather

需要收集的输入信息

Input	Description	Example
Scheduler	SLURM, PBS, LSF, local	`slurm`
Nodes/tasks/threads	Runtime layout	`2 nodes, 128 tasks, 2 threads`
GPUs	Total (whole-job) GPUs via `--gpus` , or per node via `--gpus-per-node`	`--gpus 4` or `--gpus-per-node 1`
Symptoms	Observed failure	`oom,killed,slow-gpu`
MPI/OpenMP/GPU use	Parallel modes	`mpi+openmp+gpu`
Walltime	Requested time	`12:00:00`
Scratch	Whether scratch is used	`true`

输入项	描述	示例
调度器（Scheduler）	SLURM、PBS、LSF、本地调度器	`slurm`
节点/任务/线程数	运行时布局	`2 nodes, 128 tasks, 2 threads`
GPU配置	通过 `--gpus` 指定作业总GPU数，或通过 `--gpus-per-node` 指定单节点GPU数	`--gpus 4` 或 `--gpus-per-node 1`
异常症状	观察到的失败现象	`oom,killed,slow-gpu`
MPI/OpenMP/GPU使用情况	并行模式	`mpi+openmp+gpu`
作业时长（Walltime）	请求的运行时长	`12:00:00`
临时存储（Scratch）	是否使用临时存储	`true`

Decision Guidance

决策指导

Check resource layout before changing physics settings.
Confirm module/compiler/MPI/CUDA consistency before debugging solver behavior.
Treat missing restart files and scratch cleanup as workflow failures, not physics failures.
For GPU jobs, confirm the executable was built with the requested accelerator backend.

在修改物理模拟设置前，先检查资源布局。
在调试求解器行为前，确认模块/编译器/MPI/CUDA的一致性。
将缺失重启文件和临时存储清理问题视为工作流失败，而非物理模拟失败。
对于GPU作业，确认可执行文件是基于指定的加速器后端编译的。

Script Outputs

脚本输出

scripts/hpc_runtime_doctor.py

emits:

resource_layout

(includes

tasks_per_node

total_cpus

, total

gpus

, and

gpus_per_node

)

```
diagnoses
```
```
environment_checks
```
```
retry_plan
```
```
scheduler_notes
```
```
warnings
```
(layout flags such as ranks-per-GPU oversubscription, OpenMP/thread mismatch, and uneven task placement)

In default (non-JSON) mode the script also prints the resource-layout summary, any

warnings

, environment checks, and retry plan, so the most actionable items are never hidden.

scripts/hpc_runtime_doctor.py

会输出以下内容：

resource_layout

（包含

tasks_per_node

、

total_cpus

、总

gpus

及

gpus_per_node

）

```
diagnoses
```
（诊断结果）
```
environment_checks
```
（环境检查项）
```
retry_plan
```
（重试方案）
```
scheduler_notes
```
（调度器说明）
```
warnings
```
（布局警告，如单GPU上MPI进程数过载、OpenMP/线程不匹配、任务分布不均等）

在默认（非JSON）模式下，脚本还会打印资源布局摘要、所有

warnings

、环境检查项及重试方案，确保最具操作性的信息不会被隐藏。

Workflow

工作流程

--gpus

is the total (whole-job) GPU count. Use

--gpus-per-node

(SLURM

--gres=gpu:N

semantics) when you know the per-node allocation; total GPUs are then

gpus_per_node * nodes

and it overrides

--gpus

bash

python3 skills/hpc-deployment/hpc-runtime-doctor/scripts/hpc_runtime_doctor.py \
  --scheduler slurm \
  --nodes 2 \
  --tasks 128 \
  --cpus-per-task 2 \
  --gpus 4 \
  --symptoms oom,slow-gpu \
  --uses-mpi \
  --uses-openmp \
  --uses-gpu \
  --json

The example above shares 128 ranks across 4 GPUs (32 ranks/GPU), so the

warnings

list surfaces

Many MPI ranks per GPU (32.0 ranks/GPU) may reduce GPU efficiency.

The ranks-per-GPU check uses total ranks over total GPUs, so it fires correctly on multi-node jobs (the threshold is 16 ranks/GPU).

--gpus

指定的是整个作业的GPU总数。当您知晓单节点GPU分配数时，请使用

--gpus-per-node

（符合SLURM的

--gres=gpu:N

语义）；此时总GPU数为

gpus_per_node * nodes

，且该值会覆盖

--gpus

的设置。

bash

python3 skills/hpc-deployment/hpc-runtime-doctor/scripts/hpc_runtime_doctor.py \
  --scheduler slurm \
  --nodes 2 \
  --tasks 128 \
  --cpus-per-task 2 \
  --gpus 4 \
  --symptoms oom,slow-gpu \
  --uses-mpi \
  --uses-openmp \
  --uses-gpu \
  --json

上述示例将128个进程分配到4个GPU上（每个GPU 32个进程），因此

warnings

列表会显示

Many MPI ranks per GPU (32.0 ranks/GPU) may reduce GPU efficiency.

（单GPU上MPI进程数过多（32.0个/ GPU）可能降低GPU效率）。单GPU进程数检查基于总进程数除以总GPU数，因此在多节点作业中也能正确触发（阈值为16个进程/GPU）。

Error Handling

错误处理

Invalid resource counts stop with exit code 2. Unknown symptoms are preserved as custom items for human review.

无效的资源计数会导致脚本以退出码2终止。未知症状会被保留为自定义项，供人工审核。

Limitations

局限性

This skill does not query a live scheduler. It diagnoses from the submitted layout and symptoms.

本工具不会查询实时调度器，仅基于提交的布局和症状进行诊断。

Verification checklist

验证清单

Recorded the script's
```
resource_layout
```
block and confirmed
```
tasks_per_node
```
is an integer (no fractional value) and
```
total_cpus
```
equals
```
tasks * cpus_per_task
```
; if
```
tasks_per_node
```
is fractional, the uneven-placement warning was triaged before retrying.
For GPU jobs, recorded the resolved total
```
gpus
```
(and
```
gpus_per_node
```
when set) and computed ranks/GPU =
```
tasks / gpus
```
, confirming it is at or below the 16 ranks/GPU threshold or that the resulting
```
Many MPI ranks per GPU
```
warning was deliberately accepted.
Reviewed every entry in the
```
warnings
```
list (OpenMP-with-
```
cpus_per_task=1
```
, GPU-requested-but-zero-GPUs,
```
tasks < nodes
```
, uneven placement, scratch-for-heavy-I/O) and resolved or justified each one rather than ignoring it.
Completed the
```
environment_checks
```
items as real artifacts: captured the module list, executable path/version, MPI launcher-vs-library match, accelerator build flags (CUDA/Kokkos/OpenMP), and scheduler stdout/stderr.
Mapped each observed symptom to a
```
diagnoses
```
entry and verified no symptom landed in the
```
custom
```
category unaddressed (every
```
custom
```
item had stderr/stdout/module list/command line collected for human review).
Followed the
```
retry_plan
```
: reran the smallest reproducing case, changed exactly one resource variable, enabled restart/checkpoint, and saved the scheduler script plus environment snapshot alongside the results.

记录脚本的
```
resource_layout
```
块，确认
```
tasks_per_node
```
为整数（无小数）且
```
total_cpus
```
等于
```
tasks * cpus_per_task
```
；若
```
tasks_per_node
```
为小数，需先处理任务分布不均的警告，再重试作业。
对于GPU作业，记录解析后的总
```
gpus
```
（以及设置的
```
gpus_per_node
```
），计算单GPU进程数 =
```
tasks / gpus
```
，确认该值等于或低于16个进程/GPU的阈值，或已明确接受
```
Many MPI ranks per GPU
```
警告。
审核
```
warnings
```
列表中的每一项（如OpenMP搭配
```
cpus_per_task=1
```
、请求GPU但实际分配为0、
```
tasks < nodes
```
、任务分布不均、高IO场景使用临时存储等），并逐一解决或说明理由，而非忽略。
完成
```
environment_checks
```
中的所有项：捕获模块列表、可执行文件路径/版本、MPI启动器与库的匹配情况、加速器编译标志（CUDA/Kokkos/OpenMP）及调度器的标准输出/错误输出。
将每个观察到的症状映射到
```
diagnoses
```
中的条目，确认没有症状被归入未处理的
```
custom
```
类别（每个
```
custom
```
项都已收集调度器的标准错误/输出、模块列表及命令行信息供人工审核）。
遵循
```
retry_plan
```
：运行最小复现案例，每次仅修改一个资源变量，启用重启/checkpoint功能，并将调度器脚本及环境快照与结果一起保存。

Common pitfalls & rationalizations

常见误区与合理化解释

Tempting shortcut	Why it's wrong / what to do
"It ran without crashing, so the layout is fine."	Run completion is not correctness. Review the `warnings` list and `resource_layout` -- oversubscription, uneven placement, or an idle GPU can silently slow or corrupt results without a crash.
"Per-node ranks fit the GPUs, so there's no oversubscription."	Oversubscription is total ranks over total GPUs, not per-node. The script computes `tasks / gpus` ; a multi-node job can hide a high ranks/GPU value that only the unit-consistent check exposes.
"I passed `--gpus` , so per-node GPU count doesn't matter."	`--gpus` is the whole-job total. If the cluster allocates per node, use `--gpus-per-node` (SLURM `--gres=gpu:N` ); it overrides `--gpus` and total becomes `gpus_per_node * nodes` . Mixing them up misreports ranks/GPU.
"The job was killed, so it's a physics/solver bug."	`killed` / `oom` / `timeout` are scheduler and resource categories, not physics. Check walltime, memory limits, and preemption from stdout/stderr before touching simulation parameters.
"An unknown symptom isn't in the rules, so I can skip it."	Unknown symptoms become `custom` diagnoses, not no-ops. Collect scheduler stderr/stdout, the module list, and the command line for human review -- silence is not a clean bill of health.
"Just change ranks, threads, and the build together to fix it faster."	Changing multiple variables at once makes the failure undiagnosable. The `retry_plan` mandates one variable at a time on the smallest reproducing case.

诱人的捷径	错误原因及正确做法
"作业没有崩溃，所以布局没问题。"	作业完成不代表结果正确。请审核 `warnings` 列表和 `resource_layout` ——资源过载、任务分布不均或GPU闲置可能会在不触发崩溃的情况下悄悄降低运行速度或导致结果异常。
"单节点进程数适配GPU，所以没有过载。"	资源过载是基于总进程数除以总GPU数，而非单节点。脚本会计算 `tasks / gpus` ；多节点作业可能会隐藏较高的单GPU进程数，只有基于统一单位的检查才能发现。
"我已经指定了 `--gpus` ，所以单节点GPU数不重要。"	`--gpus` 是整个作业的总GPU数。如果集群按节点分配GPU，请使用 `--gpus-per-node` （SLURM的 `--gres=gpu:N` 语义）；它会覆盖 `--gpus` 的设置，总GPU数变为 `gpus_per_node * nodes` 。混淆两者会导致单GPU进程数计算错误。
"作业被终止了，所以是物理模拟/求解器的bug。"	`killed` / `oom` / `timeout` 属于调度器和资源问题，而非物理模拟问题。在修改模拟参数前，请先检查作业时长、内存限制及抢占信息（从标准输出/错误输出中获取）。
"未知症状不在规则中，所以我可以跳过它。"	未知症状会被标记为 `custom` 诊断项，而非无操作。请收集调度器的标准错误/输出、模块列表及命令行信息供人工审核——没有报错不代表没有问题。
"同时修改进程数、线程数和编译配置，这样能更快修复问题。"	同时修改多个变量会导致无法定位失败原因。 `retry_plan` 要求在最小复现案例上每次仅修改一个变量。

Security

安全性

Input Validation

输入验证

Inputs are scalar CLI values and booleans only; there is no free-form code path.
Resource counts (
```
--nodes
```
,
```
--tasks
```
,
```
--cpus-per-task
```
,
```
--gpus
```
,
```
--gpus-per-node
```
) are validated as integers (booleans rejected), required to be non-negative and finite, and capped at 1,000,000.
```
--nodes
```
,
```
--tasks
```
, and
```
--cpus-per-task
```
must additionally be at least 1. Out-of-range, non-integer, or zero values exit with code 2.
The
```
--symptoms
```
string is capped at 64 comma-separated entries of at most 64 characters each;
```
--walltime
```
is capped at 32 characters. Oversized input exits with code 2.
Symptoms are split, trimmed, and lower-cased. Unknown symptoms are not rejected: they are preserved as
```
custom
```
diagnoses for human review.
```
--scheduler
```
is accepted as a free-form string and is not checked against an allowlist; it is only echoed back in the resource layout.

输入仅为标量CLI值和布尔值；不存在自由格式代码路径。
资源计数（
```
--nodes
```
、
```
--tasks
```
、
```
--cpus-per-task
```
、
```
--gpus
```
、
```
--gpus-per-node
```
）会被验证为整数（拒绝布尔值），要求非负且有限，上限为1,000,000。
```
--nodes
```
、
```
--tasks
```
和
```
--cpus-per-task
```
必须至少为1。超出范围、非整数或零值会导致脚本以退出码2终止。
```
--symptoms
```
字符串最多包含64个逗号分隔的条目，每个条目最多64个字符；
```
--walltime
```
最多32个字符。超出限制的输入会导致脚本以退出码2终止。
症状会被拆分、修剪并转为小写。未知症状不会被拒绝：它们会被保留为
```
custom
```
诊断项供人工审核。
```
--scheduler
```
接受自由格式字符串，不会与允许列表进行检查；仅会在资源布局中回显。

File Access

文件访问

The script reads and writes no files. All I/O is CLI args in and stdout out (indented JSON with
```
--json
```
, otherwise a human-readable summary); errors go to stderr.
Because no paths are accepted or opened, there is no filesystem traversal surface and no path-sandboxing concern.

脚本不读取或写入任何文件。所有I/O均为CLI输入和标准输出（使用
```
--json
```
时为缩进格式的JSON，否则为人类可读的摘要）；错误信息输出到标准错误。
由于不接受或打开任何路径，因此不存在文件系统遍历风险，也无需考虑路径沙箱限制。

Tool Restrictions

工具限制

allowed-tools

Read, Bash, Write, Grep, Glob

```
Bash
```
is used only to run the bundled
```
scripts/hpc_runtime_doctor.py
```
.
```
Read
```
,
```
Grep
```
, and
```
Glob
```
are used to inspect the skill's own files and any logs or submission scripts the user points at;
```
Write
```
is used to record diagnosis notes or a retry plan when asked.

allowed-tools

为

Read, Bash, Write, Grep, Glob

。

```
Bash
```
仅用于运行捆绑的
```
scripts/hpc_runtime_doctor.py
```
。
```
Read
```
、
```
Grep
```
和
```
Glob
```
用于检查工具自身的文件以及用户指定的日志或提交脚本；
```
Write
```
用于在需要时记录诊断笔记或重试方案。

Safety Measures

安全措施

No
```
eval
```
,
```
exec
```
,
```
os.system
```
, or
```
subprocess
```
; the script does not launch a scheduler or any external process and does not inspect environment variables.
Argument parsing is handled by
```
argparse
```
, and machine-readable output is emitted as JSON.
DoS exposure is bounded by the resource-count cap (1,000,000), the symptom caps (64 entries x 64 characters), and the walltime cap (32 characters).

不使用
```
eval
```
、
```
exec
```
、
```
os.system
```
或
```
subprocess
```
；脚本不会启动调度器或任何外部进程，也不会检查环境变量。
参数解析由
```
argparse
```
处理，机器可读输出以JSON格式生成。
拒绝服务（DoS）风险受限于资源计数上限（1,000,000）、症状限制（64个条目×64字符）及作业时长限制（32字符）。

References

参考资料

See
```
references/hpc_runtime_patterns.md
```
for scheduler and runtime diagnosis patterns.

调度器和运行时诊断模式请参考
```
references/hpc_runtime_patterns.md
```
。

Version History

版本历史

1.1.3: Added a Verification checklist (evidence-based items tied to
```
resource_layout
```
,
```
warnings
```
, ranks/GPU,
```
environment_checks
```
,
```
diagnoses
```
, and the
```
retry_plan
```
) and a Common pitfalls & rationalizations table.
1.1.1: Discriminating evals -- each case now pins the script's specific output (exact ranks-per-GPU warning, diagnosis categories, resource-layout fields) via deterministic
```
script_checks
```
.
1.1.0: Unit-consistent ranks-per-GPU warning (total ranks / total GPUs), new
```
--gpus-per-node
```
argument, integer
```
tasks_per_node
```
with an uneven-placement warning, full human-readable (non-JSON) output, and input caps for resource counts, symptoms, and walltime.
1.0.0: Initial HPC runtime diagnosis skill.

1.1.3：新增验证清单（与
```
resource_layout
```
、
```
warnings
```
、单GPU进程数、
```
environment_checks
```
、
```
diagnoses
```
及
```
retry_plan
```
相关的循证项）和常见误区与合理化解释表格。
1.1.1：差异化评估——每个案例现在通过确定性的
```
script_checks
```
固定脚本的特定输出（如精确的单GPU进程数警告、诊断类别、资源布局字段）。
1.1.0：基于统一单位的单GPU进程数警告（总进程数/总GPU数）、新增
```
--gpus-per-node
```
参数、整数型
```
tasks_per_node
```
及任务分布不均警告、完整的人类可读（非JSON）输出，以及资源计数、症状和作业时长的输入限制。
1.0.0：初始版本的HPC运行时诊断工具。