spark-environment-setup

Compare original and translation side by side

🇺🇸

Original

English

🇨🇳

Translation

Chinese

Spark Environment Setup

Spark环境搭建

DGX Spark ships a GB10 Grace Blackwell chip: aarch64 CPU, SM121 GPU, 128GB unified memory, CUDA 13. This is a narrower and younger platform than a standard x86 CUDA 12 box, so package selection and ABI matching matter more than usual — the wheel ecosystem for aarch64 + CUDA 13 is still filling in.

DGX Spark搭载GB10 Grace Blackwell芯片：aarch64 CPU、SM121 GPU、128GB统一内存、CUDA 13。相较于标准x86架构的CUDA 12设备，这是一个更小众、更新的平台，因此包选择和ABI匹配比通常情况更为重要——aarch64 + CUDA 13的wheel生态仍在完善中。

When to Use This Skill

适用场景

Setting up a fresh Spark box for training or inference.
Hitting an import error mentioning
```
libcudart
```
, a missing symbol, or a wheel that "installed fine but won't load."
A framework install (PyTorch, Unsloth, TRL, vLLM, xformers) fails, hangs, or silently falls back to CPU.
Deciding whether to use an NGC container or bare pip.
Restoring a working setup after an OS reinstall or a base-image update, needing to re-verify from scratch.

Each of these accepts the same general fix: match the container/wheel combination to CUDA 13 and SM121, don't fight the ABI.

为训练或推理搭建全新的Spark设备环境。
遇到提及
```
libcudart
```
的导入错误、缺失符号，或“安装成功但无法加载”的wheel问题。
框架安装（PyTorch、Unsloth、TRL、vLLM、xformers）失败、挂起或静默回退到CPU运行。
决定使用NGC容器还是原生pip安装。
操作系统重装或基础镜像更新后恢复可用环境，需要从头重新验证。

这些场景的通用解决方案是：使容器/wheel组合与CUDA 13和SM121匹配，不要强行违背ABI规则。

Container-First Rule

优先使用容器原则

Quick decision, before the detail below:

Standard training/inference work → NGC PyTorch container.
Unsloth-centric fine-tuning → Unsloth container (it ships the pinned Triton/xformers/transformers combination already validated for that path).
Neither fits (custom system package, local IDE interpreter) → bare pip, following the exact sequence further down.

Default to a container. Use

nvcr.io/nvidia/pytorch:25.09-py3

as the base for general work — the newest tag confirmed working on this hardware; pull a newer blessed tag if locally available rather than hard-blocking on

25.11-py3

. NGC's tag is dated, so running it directly is fine:

bash

docker run --runtime=nvidia --gpus all -it --rm \
  nvcr.io/nvidia/pytorch:25.09-py3

unsloth/unsloth:dgxspark-latest

is a moving tag by contrast — resolve and pin its digest before running it for anything reproducible; the bare tag is a discovery step only, not the default invocation. Full pull-inspect-pin sequence and flag rationale/volume mounts for

finetuning/

run dirs:

references/container-workflow.md

. Treat bare pip as the exception.

The reason for the container-first stance is pinning, not convenience. Triton, xformers, and transformers versions interact narrowly with GB10's SM121 target and CUDA 13; a container locks all of them together against a combination already validated on this hardware. Bare pip leaves that resolution to you, one broken import at a time.

When bare pip is warranted, follow the NVIDIA playbook's install sequence verbatim and in order:

bash

pip install "transformers==5.13.1" "peft==0.19.1" "hf_transfer==0.1.9" "datasets==4.3.0" "trl==1.8.0"
pip install --no-deps "unsloth==2026.7.2" "unsloth_zoo==2026.7.2" "bitsandbytes==0.49.2"
pip install -U "torchao==0.17.0"

The second command's

--no-deps

flag is not optional — letting pip re-resolve Unsloth's dependency tree on aarch64 is a common way to pull in an incompatible torch or triton build. The third line is not optional either: the NGC base image's bundled

torchao

is too old for current

peft

's LoRA-attach path (

ImportError: ... torchao ... only versions above 0.16.0 are supported

) — a hard blocker, not a warning. Every

==

pin above is load-bearing, taken from the dated known-good version matrix in

references/stack-matrix.md

(its

Last verified

date governs staleness) — an unpinned install resolves current PyPI versions well outside what this Unsloth release supports.

Pull a fresh tag when a new blessed release is announced. Rebuild locally from one of the two bases only when a project needs an extra system package layered in — not to "upgrade" a component the image already pins. Details on both paths:

references/container-workflow.md

One more preflight: official DGX Spark playbooks have shipped broken before. Check recent issues on

github.com/NVIDIA/dgx-spark-playbooks

(and the other resources in

references/stack-matrix.md

) before trusting a recipe verbatim for a long run.

在深入细节前的快速决策规则：

标准训练/推理工作 → 使用NGC PyTorch容器。
以Unsloth为核心的微调 → 使用Unsloth容器（它预装了经过验证的Triton/xformers/transformers固定版本组合，适配该场景）。
以上两者都不适用（自定义系统包、本地IDE解释器） → 遵循下文的精确步骤使用原生pip安装。

默认优先选择容器。通用工作负载使用

nvcr.io/nvidia/pytorch:25.09-py3

作为基础镜像——这是经确认可在该硬件上正常运行的最新标签；若本地有更新的官方认证标签，可使用更新版本，无需局限于

25.11-py3

。NGC的标签带有日期，直接运行即可：

bash

docker run --runtime=nvidia --gpus all -it --rm \
  nvcr.io/nvidia/pytorch:25.09-py3

相比之下，

unsloth/unsloth:dgxspark-latest

是一个动态更新的标签——在用于可复现的工作前，需解析并固定其摘要值；仅将裸标签用于探索阶段，而非默认调用方式。完整的拉取-检查-固定流程，以及针对

finetuning/

运行目录的参数说明/卷挂载信息，请参考

references/container-workflow.md

。将原生pip安装视为例外情况。

优先选择容器的原因是版本固定，而非便捷性。Triton、xformers和transformers的版本与GB10的SM121目标及CUDA 13的兼容性非常严格；容器会将所有组件锁定为已在该硬件上验证过的组合。而原生pip安装需要你自行解决依赖，可能会逐个遇到导入错误。

当必须使用原生pip时，请严格按照NVIDIA指南的安装顺序执行：

bash

pip install "transformers==5.13.1" "peft==0.19.1" "hf_transfer==0.1.9" "datasets==4.3.0" "trl==1.8.0"
pip install --no-deps "unsloth==2026.7.2" "unsloth_zoo==2026.7.2" "bitsandbytes==0.49.2"
pip install -U "torchao==0.17.0"

第二条命令中的

--no-deps

参数是必填项——让pip在aarch64架构上重新解析Unsloth的依赖树，很可能会引入不兼容的torch或triton构建版本。第三条命令同样必填：NGC基础镜像中预装的

torchao

版本过旧，无法支持当前

peft

的LoRA挂载路径（错误信息：

ImportError: ... torchao ... only versions above 0.16.0 are supported

）——这是硬性障碍，而非警告。上述所有

==

固定的版本都是关键，均来自

references/stack-matrix.md

中已验证的版本矩阵（其“Last verified”日期决定版本是否过时）——未固定版本的安装会解析到PyPI上的当前版本，超出该Unsloth版本支持范围。

当有新的官方认证版本发布时，拉取新标签。仅当项目需要添加额外系统包时，才基于上述两个基础镜像本地重建——不要为了“升级”镜像已固定的组件而重建。两种方式的详细说明请参考

references/container-workflow.md

。

额外的预检查：官方DGX Spark指南曾出现过错误。在长期运行中直接信任某一方案前，请查看

github.com/NVIDIA/dgx-spark-playbooks

上的最新问题（以及

references/stack-matrix.md

中的其他资源）。

The ABI Rule

ABI匹配原则

The single most common failure on Spark is a CUDA 12/13 ABI mismatch: a wheel built against

libcudart.so.12

loaded on a system that only has

libcudart.so.13

. The install usually succeeds; the failure surfaces later as a missing-symbol error or a segfault that doesn't obviously point at CUDA.

Fix: pull wheels from

download.pytorch.org/whl/cu130

(the cu130-tagged aarch64 builds), or use one of the containers above, which already carry a matched build. Before chasing a stack trace that mentions a CUDA symbol, check which CUDA tag the installed wheel was built against:

bash

python3 -c "import torch; print(torch.version.cuda)"

If that output doesn't start with

, the ABI mismatch is the first thing to fix. NGC container builds (e.g.

nvcr.io/nvidia/pytorch:25.09-py3

) build torch internally against CUDA 13 with no

+cu130

wheel tag —

pip show torch

won't say

cu130

there, and that absence alone is not a failure.

Typical symptoms:

```
ImportError: undefined symbol
```
referencing a CUDA runtime function.
A segfault on the first
```
.cuda()
```
call, no useful traceback.
A wheel that installs cleanly, then fails at import time — pip's resolver doesn't check CUDA ABI, only version constraints.
Two "identical" environments behaving differently — usually one has a cu130 wheel, the other a cu121/cu124 leftover.

The fix is the same regardless of symptom: match the wheel's CUDA tag to the system, or use a container that already does.

Spark上最常见的故障是CUDA 12/13 ABI不匹配：基于

libcudart.so.12

构建的wheel加载到仅装有

libcudart.so.13

的系统中。安装通常会成功，但故障会在后续以缺失符号错误或不明显指向CUDA的段错误形式出现。

修复方法：从

download.pytorch.org/whl/cu130

（带有cu130标签的aarch64构建包）拉取wheel，或使用上述已匹配构建版本的容器。在追踪涉及CUDA符号的堆栈跟踪前，先检查已安装wheel的CUDA标签：

bash

python3 -c "import torch; print(torch.version.cuda)"

若输出不是以

开头，首先需要修复ABI不匹配问题。NGC容器构建版本（如

nvcr.io/nvidia/pytorch:25.09-py3

）内部基于CUDA 13构建torch，没有

+cu130

的wheel标签——

pip show torch

不会显示

cu130

，这种情况本身不属于故障。

典型症状：

```
ImportError: undefined symbol
```
引用CUDA runtime函数。
首次调用
```
.cuda()
```
时出现段错误，无有效回溯信息。
wheel安装成功，但导入时失败——pip的解析器仅检查版本约束，不检查CUDA ABI。
两个“完全相同”的环境表现不同——通常一个使用cu130 wheel，另一个残留了cu121/cu124版本。

无论症状如何，修复方法都是相同的：使wheel的CUDA标签与系统匹配，或使用已完成匹配的容器。

Component Quick Table

组件状态速查表

Condensed status for the components most likely to come up. Full table with wheel URLs, build flags, the sm_121 vs sm_121a distinction, and the dated known-good version matrix:

references/stack-matrix.md

Component	Status
PyTorch	✅ official cu130 aarch64 wheels
bitsandbytes	✅ works out of the box
Triton	✅ needs the `TRITON_PTXAS_PATH` parameter set
flash-attn	❌ skip pip build; NGC bundles a working one — see `spark-training-gotchas` G2
xformers	source build only ( `TORCH_CUDA_ARCH_LIST=12.1` )
vLLM	nightly wheels only
TransformerEngine / NVFP4 train	container-only

Everything else — Unsloth, Axolotl, TRL, PEFT — installs cleanly through the container-first path above. LLaMA-Factory and NeMo are fragile on Spark; check upstream issues first.

以下是最常用组件的精简状态说明。包含wheel URL、构建参数、sm_121与sm_121a的区别，以及已验证版本矩阵的完整表格，请参考

references/stack-matrix.md

。

组件	状态
PyTorch	✅ 官方提供cu130 aarch64 wheel
bitsandbytes	✅ 开箱即用
Triton	✅ 需要设置 `TRITON_PTXAS_PATH` 参数
flash-attn	❌ 跳过pip构建；NGC容器预装可用版本——参考 `spark-training-gotchas` G2
xformers	仅支持源码构建（需设置 `TORCH_CUDA_ARCH_LIST=12.1` ）
vLLM	仅支持nightly wheel
TransformerEngine / NVFP4训练	仅支持容器方式

其他组件——Unsloth、Axolotl、TRL、PEFT——均可通过上述优先使用容器的方式顺利安装。LLaMA-Factory和NeMo在Spark上稳定性较差，使用前请先查看上游问题。

Verification Commands

验证命令

Confirm the environment can actually see the GPU before running anything expensive:

python

import torch
print(torch.cuda.is_available(), torch.version.cuda)

This call returns two values; the exact output format is one line,

<bool> <cuda-version>

text

True 13.0

If it prints

False

instead, don't jump straight to a wheel reinstall — ABI mismatch is one cause among several:

Hypothesis	Quick check
Runtime/flags	`nvidia-smi` fails in-container too
Device visibility	`echo $CUDA_VISIBLE_DEVICES`
Permissions	`ls -l /dev/nvidia*`
CUDA init state	wedged process; retry fresh shell/container
ABI mismatch (usual culprit)	`torch.version.cuda` not `13.x`

Check

nvidia-smi

first — if it doesn't show the GPU, it's one of the first three, not ABI. Reinstall a wheel only once ABI is confirmed. Per-hypothesis detail:

references/stack-matrix.md

. Run right after the container starts, before installing project-specific packages.

One more check: if Triton kernel compilation fails once training starts, set

TRITON_PTXAS_PATH=/usr/local/cuda/bin/ptxas

and retry — see

references/stack-matrix.md

for the full workaround list.

在运行高负载任务前，先确认环境能否识别GPU：

python

import torch
print(torch.cuda.is_available(), torch.version.cuda)

该调用返回两个值，输出格式为一行：

<bool> <cuda-version>

：

text

True 13.0

若输出为

False

，不要直接重新安装wheel——ABI不匹配只是原因之一：

假设原因	快速检查方法
运行时/参数	容器内 `nvidia-smi` 也失败
设备可见性	`echo $CUDA_VISIBLE_DEVICES`
权限	`ls -l /dev/nvidia*`
CUDA初始化状态	进程阻塞；尝试重新打开shell/容器
ABI不匹配（常见原因）	`torch.version.cuda` 不是 `13.x`

首先检查

nvidia-smi

——如果无法显示GPU，原因属于前三者，而非ABI问题。仅在确认ABI匹配后，再重新安装wheel。各原因的详细说明请参考

references/stack-matrix.md

。请在容器启动后、安装项目专属包前运行该检查。

额外检查：若训练开始后Triton内核编译失败，设置

TRITON_PTXAS_PATH=/usr/local/cuda/bin/ptxas

并重试——完整的解决方案列表请参考

references/stack-matrix.md

。

Next Steps

后续步骤

A verified environment is only the starting point. See also:

spark-training-gotchas

for failure preflights before a training run, and

spark-memory-thermal-ops

for unified-memory OOMs and thermal throttling during long ones.

验证完成的环境只是起点。训练运行前的故障预检查请参考

spark-training-gotchas

；长时间运行中统一内存OOM和热节流问题请参考

spark-memory-thermal-ops

。