spark-memory-thermal-ops

Compare original and translation side by side

🇺🇸

Original

English

🇨🇳

Translation

Chinese

Spark Memory & Thermal Ops

Spark内存与散热运维

DGX Spark's GB10 chip has one 128GB unified memory (UMA) pool shared by CPU and GPU, and a sustained power ceiling well below its rated figure. Both break discrete-GPU assumptions: headroom isn't what

nvidia-smi

reports, and a run that starts fast will slow down mid-job with nothing misconfigured. This skill covers planning memory headroom, working an actual OOM, and watching thermals across a long job. For launch-time failure modes (ABI mismatches, flash-attn, playbook breakage), see

spark-training-gotchas

— this skill assumes the job starts.

DGX Spark的GB10芯片拥有一个由CPU和GPU共享的128GB统一内存（UMA）池，其持续功耗上限远低于额定值。这两点打破了独立GPU的固有假设：内存余量并非

nvidia-smi

所报告的数值，且初始运行快速的作业可能在中途变慢，却不存在配置错误。本技能涵盖内存余量规划、实际OOM问题处理，以及长时间作业中的散热监控。若遇到启动阶段的故障模式（ABI不匹配、flash-attn问题、脚本执行失败），请参阅

spark-training-gotchas

——本技能假定作业已成功启动。

Common Issues Quick Reference

常见问题速查

Situation	Do this
Planning headroom before launch	Budget against `free -g` , not `nvidia-smi` — see UMA Memory Model
Job OOMs on unified memory	Work the OOM Ladder in order: flush, then batch/pack, then method downgrade
Throughput drops mid-run	Check the power/temp log before assuming a config bug — see Thermal Monitoring
Trainer + inference server both wanted	Run one at a time — see Concurrent Workloads

场景	操作建议
启动前规划内存余量	以 `free -g` 的结果为预算依据，而非 `nvidia-smi` ——详见UMA内存模型
作业因统一内存发生OOM	按顺序执行OOM排查步骤：先刷新缓存，再调整批次/打包长度，最后降级训练方法
运行中途吞吐量下降	先检查功耗/温度日志，再假设是配置问题——详见散热监控
需同时运行训练器与推理服务器	一次仅运行一个——详见并发工作负载

When to Use This Skill

适用场景

Sizing a training run against the 128GB pool before launch — will this model, method, and batch/pack combination fit.
A run OOMs mid-load or mid-step and the remediation order matters — what to try first, second, third.
Watching temperature and power during a multi-hour job, deciding whether a slowdown is thermal throttling or something else.
Planning to run a trainer alongside an inference server (vLLM, Ollama) on the same box.

启动前针对128GB内存池规划训练作业规模——判断模型、训练方法及批次/打包组合是否能容纳。
作业在加载或运行步骤中发生OOM，且修复顺序至关重要——确定优先尝试的方案。
在数小时的作业过程中监控温度与功耗，判断性能下降是否由热节流导致。
计划在同一设备上同时运行训练器与推理服务器（vLLM、Ollama）。

UMA Memory Model

UMA内存模型

Spark has no separate GPU VRAM — the GPU and CPU share one 128GB pool. Two consequences:

nvidia-smi
and
cudaMemGetInfo
underreport pressure — or report nothing at all. Both report CUDA-allocator-visible memory, not the pool's actual state — a box can show headroom in
```
nvidia-smi
```
and still OOM, because page-cache and mmap'd pages the allocator doesn't see consume the same pool. On some driver/setups, the memory query returns
```
[N/A], [N/A]
```
outright instead of a number — a script grepping for a numeric value there gets nothing, not a misleading undercount (see
```
spark-training-gotchas
```
gotcha G3).
Model load is a transient peak, not the steady state. Loading safetensors weights mmaps the file, then copies into CUDA tensors — for a window during load, both the mmap'd pages and the CUDA copy count against the pool at once. A model that fits while training can still OOM during load if headroom was sized for the post-load footprint instead of this doubled transient.

Plan and diagnose with

free -g

, not

nvidia-smi

bash

free -g | awk 'NR==2 {print "free:", $4, "GB"}'

Rule of thumb: take that free figure, subtract a few GB for OS/driver overhead, and budget against the result — not the 128GB spec number. The worksheet in

references/uma-accounting.md

accepts parameter count, dtype, and method as input, and returns a memory estimate to compare against known anchors.

Spark没有独立的GPU显存——GPU和CPU共享一个128GB的内存池。带来两个影响：

nvidia-smi
和
cudaMemGetInfo
会低估内存压力，甚至完全无法报告。两者仅能报告CUDA分配器可见的内存，而非内存池的实际状态——设备可能在
```
nvidia-smi
```
显示有剩余内存的情况下仍发生OOM，因为页缓存和内存映射页（分配器无法识别）会占用同一内存池。在部分驱动/配置下，内存查询会直接返回
```
[N/A], [N/A]
```
而非具体数值——若脚本在此处提取数值，将无法得到结果，而非误导性的低估数值（详见
```
spark-training-gotchas
```
中的问题G3）。
模型加载是瞬时峰值，而非稳态。 加载safetensors权重时会先映射文件，再复制到CUDA张量——在加载的某个窗口期，内存映射页和CUDA副本会同时占用内存池。即使训练时模型能容纳，若内存余量是按加载后的占用量规划，仍可能在加载阶段发生OOM。

请使用

free -g

而非

nvidia-smi

进行规划与诊断：

bash

free -g | awk 'NR==2 {print "free:", $4, "GB"}'

经验法则：取该空闲数值，减去几GB的操作系统/驱动开销，以此作为预算依据——而非128GB的标称值。

references/uma-accounting.md

中的工作表可输入参数数量、数据类型和训练方法，返回内存估算值，以便与已知基准对比。

Planning Sequence

规划流程

Before launch, work through these in order:

Read
```
free -g
```
; subtract OS/driver overhead for the budget.
Estimate weights + optimizer + gradients + activations from
```
references/uma-accounting.md
```
.
Compare against the closest anchor (70B QLoRA, 27B LoRA, 9B full FT), not the estimate alone.
If the estimate is close to the budget, start with shorter packing or a smaller batch — cheaper than hitting the OOM Ladder mid-run.

启动前，请按以下顺序操作：

查看
```
free -g
```
结果；减去操作系统/驱动开销得到预算值。
通过
```
references/uma-accounting.md
```
估算权重+优化器+梯度+激活值的内存占用。
将估算值与最接近的基准（70B QLoRA、27B LoRA、9B全量微调）对比，而非仅依赖估算值。
若估算值接近预算，先从缩短打包长度或减小批次大小入手——比运行中途触发OOM排查步骤成本更低。

Example: Sizing a 70B QLoRA Run

示例：70B QLoRA作业规模规划

A sanity check of the worksheet formula against the ≈40GB anchor:

python

params = 70e9
weights_gb = params * 0.5 / 1e9      # NF4, step 1
adapter_gb = 0.5                     # step 5, negligible
total_gb = weights_gb + adapter_gb   # + activations
print(f"{total_gb:.0f}GB before activations")

Weights alone land near the ≈40GB anchor — a plan estimating far above that for the same model class is a signal to recheck dtype and method.

通过工作表公式与≈40GB基准进行合理性校验：

python

params = 70e9
weights_gb = params * 0.5 / 1e9      # NF4，步骤1
adapter_gb = 0.5                     # 步骤5，可忽略
total_gb = weights_gb + adapter_gb   # + 激活值
print(f"{total_gb:.0f}GB before activations")

仅权重的占用量就接近≈40GB基准——若针对同类模型的规划远高于此，需重新检查数据类型和训练方法。

The OOM Ladder

OOM排查步骤

When a job OOMs on unified memory, work this ladder in order. Each step is more disruptive than the last — don't skip ahead: reducing batch size is never step 1.

Flush the buffer cache. Page cache from a previous run or a large dataset read often accounts for GB of the "missing" headroom. This costs nothing but a rerun and doesn't touch the job's configuration:
bash
```
sync; echo 3 > /proc/sys/vm/drop_caches
```
Needs root; a between-run reset, not a mid-training step. See
```
spark-training-gotchas
```
(gotcha G3) for the full diagnostic behind this step.
Reduce batch size or packing length. Only after a flush fails to free enough headroom, cut batch size or packing length — the first step that changes what the run does. Prefer packing length first; it drives activation footprint more directly at long context.
Downgrade the method: bf16 LoRA before QLoRA. If flushing and shrinking batch/pack still OOM, drop the method a tier — bf16 LoRA is next, not the reverse. QLoRA's bitsandbytes dequantization buffers are transient CUDA-side allocations that can OOM before an equivalent bf16 LoRA run would, even though QLoRA's steady-state footprint is smaller. A QLoRA OOM is not proof the model doesn't fit.

Fall back further (smaller model, multi-Spark) only after all three steps and the job still won't fit.

当作业因统一内存发生OOM时，请按以下顺序执行步骤。每一步的破坏性都比前一步更强——请勿跳过：减小批次大小永远不是第一步。

刷新缓冲区缓存。 前一次运行的页缓存或大型数据集读取通常会占用数GB的“缺失”内存余量。此操作仅需重新运行作业，无需修改配置：
bash
```
sync; echo 3 > /proc/sys/vm/drop_caches
```
需要root权限；仅能在两次作业之间执行，无法在训练中途操作。详见
```
spark-training-gotchas
```
（问题G3）中该步骤的完整诊断说明。
减小批次大小或打包长度。 仅在刷新缓存无法释放足够内存余量后，再调整批次大小或打包长度——这是首个会改变作业行为的步骤。优先调整打包长度；在长上下文场景下，它对激活值占用的影响更直接。
降级训练方法：优先使用bf16 LoRA而非QLoRA。 若刷新缓存和调整批次/打包长度后仍发生OOM，将训练方法降级一级——应选择bf16 LoRA，而非反向操作。QLoRA的bitsandbytes反量化缓冲区是瞬时CUDA端分配，可能在等效bf16 LoRA作业之前就触发OOM，尽管QLoRA的稳态内存占用更小。QLoRA发生OOM并不代表模型无法容纳。

只有在完成以上三步后作业仍无法容纳时，才考虑进一步降级（更小模型、多Spark节点）。

Thermal Monitoring

散热监控

Multi-hour runs push into Spark's sustained power ceiling, well under the rated figure — expected platform behavior, not a symptom to explain away:

Sample temperature and power alongside the training logs, not after a slowdown is noticed — every 30-60 seconds correlates a throughput drop with a thermal event. Keep the CSV output format
```
assets/thermal-sample.sh
```
writes, so timestamps line up against the log:
bash
```
bash assets/thermal-sample.sh 30 thermal.log
```
A sustained ~100W power draw is the platform cap, not a configuration bug. Don't re-tune batch size or precision to "fix" a plateau that's the box behaving normally under load. If temperature climbs while power stays flat under the rated 240W figure, that's the signature to recognize.
Log throttle events explicitly instead of letting a run silently slow down unrecorded. A run whose per-step time doubles two hours in should show that in the log, correlated against the thermal sample at that timestamp. Full throttling diagnostics:
```
spark-training-gotchas
```
(gotcha G4).

数小时的作业会触及Spark的持续功耗上限，该上限远低于额定值——这是平台的正常行为，而非需要排查的异常：

同步采样温度和功耗与训练日志，而非在性能下降后才检查——每30-60秒采样一次，将吞吐量下降与热事件关联起来。请保留
```
assets/thermal-sample.sh
```
输出的CSV格式，以便时间戳与日志对齐：
bash
```
bash assets/thermal-sample.sh 30 thermal.log
```
持续≈100W的功耗是平台上限，而非配置错误。 无需调整批次大小或精度来“修复”这种平台负载下的正常性能平稳期。若温度上升但功耗保持在额定240W以下的平稳状态，这是需要识别的典型特征。
明确记录节流事件，避免作业在无记录的情况下静默变慢。若作业在运行两小时后单步时间翻倍，应在日志中体现，并与对应时间戳的散热采样数据关联。完整的节流诊断请参阅
```
spark-training-gotchas
```
（问题G4）。

Concurrent Workloads

并发工作负载

Because the 128GB pool is global, eviction happens without either process's logs showing an OOM:

The one-heavy-job rule applies to uncapped or near-capacity workloads — an uncapped trainer and inference server (vLLM, Ollama) compete for the same pool. A small, capped workload doesn't: a <4GB LoRA fine-tune coexists fine alongside vLLM capped at
```
gpu-memory-utilization<=0.5
```
— check the other process's cap, not just its presence, before stopping it.
Inference servers evict trainer pages silently under uncapped/near-capacity contention, and vice versa — neither logs an error, so a slow run or lost KV cache is a contention symptom to check for. Stop unrelated uncapped servers before a long or full-pool run.

Check for GPU-resident processes first:

bash

ps aux | grep -E 'vllm|ollama|trl|axolotl' | grep -v grep

This procedure complements

spark-training-gotchas

(gotchas G3, G4, G6) — that skill covers launch-time failures; this one, the running job.

Memory math worksheets:

references/uma-accounting.md

由于128GB内存池是全局共享的，内存回收会在无任何进程日志显示OOM的情况下发生：

单重作业规则适用于无限制或接近满负载的工作负载——无限制的训练器与推理服务器（vLLM、Ollama）会竞争同一内存池。小型、有限制的工作负载则不受此影响：<4GB的LoRA微调可与限制为
```
gpu-memory-utilization<=0.5
```
的vLLM共存——在停止其他进程前，请先检查其内存限制，而非仅看进程是否存在。
在无限制/接近满负载的竞争场景下，推理服务器会静默回收训练器的内存页，反之亦然——两者均不会记录错误，因此运行缓慢或KV缓存丢失是需要排查的竞争症状。在运行长时间或满负载作业前，请停止无关的无限制服务器。

首先检查GPU驻留进程：

bash

ps aux | grep -E 'vllm|ollama|trl|axolotl' | grep -v grep

本流程是

spark-training-gotchas

（问题G3、G4、G6）的补充——该技能涵盖启动阶段故障，而本技能针对运行中的作业。

内存计算工作表：

references/uma-accounting.md

。