databricks-ai-runtime
Compare original and translation side by side
🇺🇸
Original
English🇨🇳
Translation
ChineseDatabricks AI Runtime (databricks air
)
databricks airDatabricks AI Runtime (databricks air
)
databricks airDatabricks AI Runtime is a set of CLI commands for submitting GPU training workloads to Databricks serverless compute. It manages environment setup, distributed training configuration, and workload lifecycle, without requiring you to manage clusters or infrastructure.
databricks airThis skill covers the three everyday flows: starting a run, checking on a run, and monitoring a run as it trains.
Databricks AI Runtime 是一组 CLI 命令,用于将 GPU 训练工作负载提交到 Databricks serverless compute。它负责管理环境设置、分布式训练配置和工作负载生命周期,无需你管理集群或基础设施。
databricks air本 skill 涵盖三个日常流程:启动运行、检查运行以及在训练期间监控运行。
Reference discipline
参考原则
The CLI's built-in help is the source of truth for field names, accelerator
types, defaults, and constraints. Prefer it at runtime over anything memorized.
Look up a config field by passing its path to on the command (the
path is a separate argument):
-hrunbash
databricks air run -h config # list the YAML fields
databricks air run -h config.compute # one field
databricks air run -h config.compute.accelerator_type # a nested fielddatabricks air <subcommand> --helpCLI 的内置帮助信息是字段名、加速器类型、默认值和约束条件的事实来源。运行时,应优先使用它,而不是依赖记忆中的内容。
若要查找某个配置字段,请将它的路径作为单独参数传给 命令的 选项:
run-hbash
databricks air run -h config # list the YAML fields
databricks air run -h config.compute # one field
databricks air run -h config.compute.accelerator_type # a nested fielddatabricks air <subcommand> --helpSession setup
会话设置
Check the CLI version, confirm the command is available, and list your
Databricks config profiles:
bash
databricks version
databricks air --help
databricks auth profiles --skip-validate # available Databricks config profilesThe examples use flattened fields, available in
Databricks CLI 1.20.0 and newer.
On an older CLI, use the equivalent nested snapshot format shown under
Workload YAML. CLI 1.20.0 and newer accept both formats.
Do not mix nested and flattened fields in the same configuration.
code_sourcePass on every command. Add for scripted/programmatic
calls so you get a structured envelope instead of human text:
-p <profile>-o json- Success:
{"v": 1, "ts": "...", "data": {...}} - Error:
{"v": 1, "ts": "...", "error": {"code": "...", "kind": "...", "message": "...", "retryable": bool}}
Config validation (input) errors are the exception: they print as plain text to
stderr with a non-zero exit even under , so check the exit code and
stderr too, not just the parsed JSON. When you do get a structured error, its
tells you what to do:
-o jsonkind| kind | Meaning | Action |
|---|---|---|
| Not authenticated / permission denied / feature disabled / non-retryable | If auth: |
| Bad run ID / resource | Check the input |
| Idempotency key collision | Use a new |
| Retryable | Retry |
检查 CLI 版本,确认命令可用,并列出你的 Databricks 配置档案:
bash
databricks version
databricks air --help
databricks auth profiles --skip-validate # available Databricks config profiles示例使用扁平化的 字段,该形式适用于
Databricks CLI 1.20.0 及更高版本。
对于较旧的 CLI,请使用 Workload YAML 下所示的等效嵌套 snapshot 格式。Databricks CLI 1.20.0 及更高版本接受这两种格式。
不要在同一配置中混用嵌套字段和扁平化字段。
code_source每个命令都传入 。对于脚本化或编程式调用,添加 ,这样获得的是结构化封装,而不是人类可读的文本:
-p <profile>-o json- 成功:
{"v": 1, "ts": "...", "data": {...}} - 错误:
{"v": 1, "ts": "...", "error": {"code": "...", "kind": "...", "message": "...", "retryable": bool}}
配置验证(输入)错误是例外:即使在 下,也会以纯文本形式输出到 stderr 并返回非零退出码,因此除了解析后的 JSON,还要检查退出码和 stderr。获得结构化错误时,其 会告诉你应采取什么操作:
-o jsonkind| kind | 含义 | 操作 |
|---|---|---|
| 未认证、权限被拒、功能未启用或不可重试 | 如果是身份验证问题: |
| 运行 ID 或资源无效 | 检查输入 |
| Idempotency key 冲突 | 使用新的 |
| 可重试 | 重试 |
Subcommands
子命令
| Command | What it does |
|---|---|
| Submit a workload from a YAML file. Add |
| Show status, config, timing, and the failure reason for one run. |
| List your active runs. A run that just finished drops out of the default view; add |
| Cancel one or more runs, or all of your active runs ( |
| Stream a running run's logs, or fetch a finished run's logs. |
| View the provisioned GPU pools available to the workspace. |
| Push a local or remote container image to Databricks Artifact Registry. See docker-images.md. |
| Convert a workload YAML into a Databricks Asset Bundle (local only; deploy with |
For flags and defaults, run .
databricks air <subcommand> --help| 命令 | 作用 |
|---|---|
| 从 YAML 文件提交工作负载。添加 |
| 显示某次运行的状态、配置、耗时和失败原因。 |
| 列出你的活动运行。刚完成的运行会从默认视图中消失;添加 |
| 取消一个或多个运行,或取消你的所有活动运行( |
| 流式传输正在运行的日志,或获取已结束运行的日志。 |
| 查看工作区中可用的预配 GPU 池。 |
| 将本地或远程容器镜像推送到 Databricks Artifact Registry。参见 docker-images.md。 |
| 将工作负载 YAML 转换为 Databricks Asset Bundle(仅在本地执行;使用 |
有关标志和默认值,请运行 。
databricks air <subcommand> --helpWorkload YAML
工作负载 YAML
yaml
experiment_name: my-training-job # alphanumeric, - and _ only
compute:
num_accelerators: 1
accelerator_type: GPU_1xA10 # run `databricks air run -h config.compute` for valid values
environment:
version: "6" # Standard environment; use databricks_ai_v6 for PyTorch preinstalled
dependencies:
- torch
- mlflow
code_source:
root_path: "." # relative to workload.yaml; extracted to $CODE_SOURCE_PATH on each node
max_retries: 0 # while iterating; see Retries below
command: |-
cd "$CODE_SOURCE_PATH"
python train.pySubmit with . Field
details live in (the authoritative
source).
databricks air run --file workload.yaml -p <profile>databricks air run -h config.<field>For an older CLI, replace only the block above with:
code_sourceyaml
code_source:
type: snapshot
snapshot:
root_path: "."The values and behavior are the same: add under
and put all snapshot fields (, and any , , or
) inside . Keep the rest of the workload
unchanged. Check for the fields
supported by the installed CLI.
type: snapshotcode_sourceroot_pathremote_volumegitinclude_pathscode_source.snapshotdatabricks air run -h config.code_sourceyaml
experiment_name: my-training-job # alphanumeric, - and _ only
compute:
num_accelerators: 1
accelerator_type: GPU_1xA10 # run `databricks air run -h config.compute` for valid values
environment:
version: "6" # Standard environment; use databricks_ai_v6 for PyTorch preinstalled
dependencies:
- torch
- mlflow
code_source:
root_path: "." # relative to workload.yaml; extracted to $CODE_SOURCE_PATH on each node
max_retries: 0 # while iterating; see Retries below
command: |-
cd "$CODE_SOURCE_PATH"
python train.py使用 提交。字段详情位于 (权威来源)。
databricks air run --file workload.yaml -p <profile>databricks air run -h config.<field>对于较旧的 CLI,只需将上面的 块替换为:
code_sourceyaml
code_source:
type: snapshot
snapshot:
root_path: "."这些值和行为完全相同:在 下添加 ,并将所有 snapshot 字段(,以及可选的 、 或 )放在 内。其余工作负载保持不变。运行 ,查看已安装 CLI 支持的字段。
code_sourcetype: snapshotroot_pathremote_volumegitinclude_pathscode_source.snapshotdatabricks air run -h config.code_sourceAccelerator types
加速器类型
Run to see the accelerator types. On CLI
1.23.0 and newer it lists the types available in the current workspace and
how many GPUs each type has per node; if it cannot reach the workspace, it
prints a warning and falls back to a built-in list. Do not guess a type from
memory: availability depends on the workspace, cloud, and region, and matching
is case-sensitive.
databricks air run -h config.computeType names follow , for example ,
, , and . is the total
GPU count across all nodes, so it must be a multiple of the per-node count: 8
on an type is one node, 16 is two nodes. Only the types support
multi-GPU and multi-node training.
GPU_<GPUs per node>x<GPU>GPU_1xA10GPU_1xH100GPU_8xH100GPU_8xB300num_accelerators8x8x运行 查看加速器类型。在 CLI
1.23.0 及更高版本中,它会列出当前工作区中可用的类型以及每种类型在每个节点上的 GPU 数量;如果无法连接工作区,则会打印警告并回退到内置列表。不要凭记忆猜测类型:可用性取决于工作区、云和区域,且匹配区分大小写。
databricks air run -h config.compute类型名称遵循 ,例如 、
、 和 。 是所有节点的 GPU 总数,因此它必须是每节点数量的倍数:在 类型上,8 表示一个节点,16 表示两个节点。只有 类型支持多 GPU 和多节点训练。
GPU_<每节点 GPU 数量>x<GPU>GPU_1xA10GPU_1xH100GPU_8xH100GPU_8xB300num_accelerators8x8xPython environment
Python 环境
Pick the environment from what the training code imports, before you submit,
so a missing package does not surface only after a GPU has been provisioned:
| What it provides | Use when |
|---|---|---|
| Standard environment: Python and CUDA only. Since version 5, it does not include | You control the dependency stack. List every import, including |
| Databricks AI environment: PyTorch, Transformers, and other ML frameworks preinstalled. | Typical PyTorch or Hugging Face training. Add extras under |
| Your own Docker image. | System libraries or CUDA extensions; see Custom Docker images. |
Before submitting, check the script's imports against the chosen environment.
For the exact packages in each environment version, see the
environment version release notes.
Do not combine with or .
unity_catalog_imageversiondependencies提交前,应根据训练代码导入的内容选择环境,避免等到 GPU 已预配后才发现缺少包:
| 提供的内容 | 使用场景 |
|---|---|---|
| 标准环境:仅包含 Python 和 CUDA。从版本 5 开始,它不包含 | 你自行控制依赖栈。将所有 import,包括 |
| Databricks AI 环境:预装 PyTorch、Transformers 和其他 ML 框架。 | 典型的 PyTorch 或 Hugging Face 训练。在 |
| 你自己的 Docker 镜像。 | 需要系统库或 CUDA 扩展;参见自定义 Docker 镜像。 |
提交前,将脚本中的 import 与所选环境进行核对。每个环境版本中的确切软件包,请参阅环境版本发布说明。
不要将 与 或 结合使用。
unity_catalog_imageversiondependenciesConfiguring the workload
配置工作负载
These optional top-level fields cover the common needs. Run
for the full reference on any of them.
databricks air run -h config.<field>以下可选的顶层字段涵盖了常见需求。对于其中任何一个字段,运行 即可获取完整参考。
databricks air run -h config.<field>Hugging Face gated models
Hugging Face 受限模型
When a model needs a Hugging Face (HF) token, lead with this checklist before
asking for other training details or running secrets commands:
- Accept the model license on HF and, if required, wait for access approval. Create a fine-grained read-only token with Read access to contents of all public gated repos you can access. For a private repo, also grant read access to that specific repo.
- Choose an existing scope the user owns and can write. If they need to find
one, have them filter to their own username or scopes they created, then confirm ownership and access; a name match alone is not proof. For example, the user can run the following with their own username:
list-scopesNever browse or inspect other users' scopes or keys. Create a scope only if needed; ifbashdatabricks secrets list-scopes -p <profile> -o json | jq -r --arg user "<your-username>" '.[] | select(.name | contains($user)) | .name'reportscreate-scope, reuse an existing user-owned scope instead. The user may need to runMax limit of 2000 scopes has been reachedon their own scope to check access. Before usinglist-aclsin any<scope>command, validate it: it must be an existing scope the user owns, from their own scopes, and match the scope-name charsetdatabricks secrets .... If it doesn't, have the user run the commands themselves. Always double-quote it. If the scope is Azure Key Vault-backed (^[A-Za-z0-9_.@-]+$isbackend_type),AZURE_KEYVAULTcan't write to it: have the user add the token to that vault with their own tooling and use that secret's name as the key below.put-secret - First check whether already exists in the scope, using the key-name check in step 4. If it does, reuse it only if it is the right token; otherwise pick a new key such as
hf-token, or get the user's explicit OK before overwriting it, since other workloads may read it. Use the chosen key in steps 4 and 5. Have the user run this in their own terminal and enter the token at the hidden prompt. Never paste the token into chat or a command line.hf-token-gemmabashdatabricks secrets put-secret "<scope>" hf-token -p <profile> - Once they have stored it and named their scope, verify only the key name in
that one scope, filtering the output to :
hf-tokenbashdatabricks secrets list-secrets "<scope>" -p <profile> -o json | jq -r '.[] | select(.key == "hf-token") | .key' - Reference in the workload. Quote the reference, particularly when the scope contains
secrets: {HF_TOKEN: "<scope>/hf-token"}or@. Transformers reads.automatically.HF_TOKEN
Gemma 3 is gated; Mistral Small 3.2, gpt-oss-20b, and Qwen3 are ungated and do
not need a token just to download their public weights. Check the specific
model's HF page for current access requirements.
当模型需要 Hugging Face (HF) token 时,在询问其他训练详情或运行 secrets 命令之前,先按照以下清单操作:
- 在 HF 上接受模型许可证;如需审批,则等待访问权限获批。创建一个 fine-grained 只读 token,并授予 Read access to contents of all public gated repos you can access。对于私有仓库,还需授予对该特定仓库的读取权限。
- 选择一个用户拥有且可写的现有 scope。如果用户需要查找,可让用户运行 ,按自己的用户名或创建的 scopes 进行筛选,然后确认所有权和访问权限;仅名称匹配并不能证明所有权。例如,用户可以使用自己的用户名运行以下命令:
list-scopes绝不要浏览或检查其他用户的 scopes 或 keys。仅在确有需要时创建 scope;如果bashdatabricks secrets list-scopes -p <profile> -o json | jq -r --arg user "<your-username>" '.[] | select(.name | contains($user)) | .name'报告create-scope,则改为复用现有的用户自有 scope。用户可能需要在自己的 scope 上运行Max limit of 2000 scopes has been reached以检查访问权限。在任何list-acls命令中使用databricks secrets ...前,先进行验证:它必须是用户从自己的 scopes 中选取且拥有的现有 scope,并符合 scope 名称字符集<scope>。如果不符,应让用户自行运行这些命令。始终使用双引号包裹它。如果该 scope 由 Azure Key Vault 支撑(^[A-Za-z0-9_.@-]+$为backend_type),AZURE_KEYVAULT将无法向其写入:应让用户使用自己的工具将 token 添加到该 vault,并把该 secret 的名称用作下面的 key。put-secret - 使用第 4 步的键名检查方式,先检查该 scope 中是否已存在 。如果存在,仅在它确实是所需 token 时复用;否则请选择新 key,例如
hf-token,或者在覆盖前获得用户的明确同意,因为其他工作负载可能读取该 key。在第 4 步和第 5 步中使用选定的 key。让用户在自己的终端中运行以下命令,并在隐藏输入提示中输入 token。绝不要将 token 粘贴到聊天或命令行中。hf-token-gemmabashdatabricks secrets put-secret "<scope>" hf-token -p <profile> - 用户存储 token 并提供 scope 名称后,仅验证该 scope 中的键名,并将输出筛选为 :
hf-tokenbashdatabricks secrets list-secrets "<scope>" -p <profile> -o json | jq -r '.[] | select(.key == "hf-token") | .key' - 在工作负载中引用 。请为引用添加引号,尤其是在 scope 包含
secrets: {HF_TOKEN: "<scope>/hf-token"}或@时。Transformers 会自动读取.。HF_TOKEN
Gemma 3 属于受限模型;Mistral Small 3.2、gpt-oss-20b 和 Qwen3 无访问限制,仅下载公开权重不需要 token。请查看特定模型的 HF 页面以了解当前的访问要求。
Environment variables and secrets
环境变量和密钥
Set plain values under ; pull sensitive values from Databricks
Secrets under (each entry maps an env var name to a , and
the value is materialized on the worker, never logged):
env_variablessecretsscope/keyyaml
env_variables:
BATCH_SIZE: "32"
secrets:
HF_TOKEN: "<scope>/hf-token" # -> $HF_TOKEN on every node
WANDB_API_KEY: "<scope>/wandb_key"The scope and keys must already exist. Prefer a user-owned scope; create one
only if needed, and have the user store secret values in their own terminal.
Secret env var names must not collide with names.
env_variables在 下设置普通值;从 Databricks Secrets 中获取敏感值并放入 (每个条目将环境变量名映射到 ,值会在 worker 上实例化,且永不写入日志):
env_variablessecretsscope/keyyaml
env_variables:
BATCH_SIZE: "32"
secrets:
HF_TOKEN: "<scope>/hf-token" # -> $HF_TOKEN on every node
WANDB_API_KEY: "<scope>/wandb_key"scope 和 key 必须已经存在。优先使用用户自有 scope;仅在需要时创建,并让用户在自己的终端中存储 secret 值。Secret 环境变量名不得与 中的名称冲突。
env_variablesParameters (hyperparameters)
参数(超参数)
parameters$HYPERPARAMETERS_PATHyaml
parameters:
learning_rate: 0.001
batch_size: 32
warmup_steps: 1000
use_amp: trueRead them in the script:
python
import os, yaml
with open(os.environ["HYPERPARAMETERS_PATH"]) as f:
hp = yaml.safe_load(f)
lr = hp["learning_rate"]Values can be strings, numbers, booleans, lists, or nested dicts. To sweep a
hyperparameter without editing the YAML, resubmit with
(and a fresh idempotency key).
--override parameters.learning_rate=0.0005parameters$HYPERPARAMETERS_PATHyaml
parameters:
learning_rate: 0.001
batch_size: 32
warmup_steps: 1000
use_amp: true在脚本中读取它们:
python
import os, yaml
with open(os.environ["HYPERPARAMETERS_PATH"]) as f:
hp = yaml.safe_load(f)
lr = hp["learning_rate"]值可以是字符串、数字、布尔值、列表或嵌套字典。要在不编辑 YAML 的情况下扫描某个超参数,请使用 重新提交(并使用新的 idempotency key)。
--override parameters.learning_rate=0.0005Retries
重试
Failed attempts are retried automatically: defaults to 3 when
unset (confirm with ). Each retry
provisions GPUs and reruns the whole command, so a deterministic failure (an
import error, a typo, a bad path) fails 4 times before the run gives up.
max_retriesdatabricks air run -h config.max_retriesSet while writing, debugging, or iterating on a workload,
either in the YAML or with . Raise it only for a
long, stable training run where retrying can recover from transient
infrastructure failures, and make the training code resume from a checkpoint
so a retry does not start over. When is set, it applies
per attempt.
max_retries: 0--override max_retries=0timeout_minutes失败的尝试会自动重试:未设置时, 默认为 3(通过 确认)。每次重试都会重新预配 GPU 并重新运行整个命令,因此确定性失败(import 错误、拼写错误、错误路径)会在任务放弃前失败 4 次。
max_retriesdatabricks air run -h config.max_retries编写、调试或迭代工作负载时,将 设为 0,无论是在 YAML 中设置,还是使用 。仅当长时间且稳定的训练任务可以通过重试从暂时性基础设施故障中恢复时,才提高该值;同时应确保训练代码可从 checkpoint 恢复,避免重试时从头开始。设置 时,它适用于每次尝试。
max_retries--override max_retries=0timeout_minutesPermissions
权限
By default only you can see the run. Grant others access to the submitted job
with (DABs-compatible format); levels are ,
, :
permissionsCAN_VIEWCAN_MANAGE_RUNCAN_MANAGEyaml
permissions:
- group_name: data-engineering
level: CAN_VIEW
- user_name: alice@example.com
level: CAN_MANAGE默认情况下,只有你能看到该运行。使用 (兼容 DABs 的格式)授予他人访问所提交作业的权限;权限级别为 、、:
permissionsCAN_VIEWCAN_MANAGE_RUNCAN_MANAGEyaml
permissions:
- group_name: data-engineering
level: CAN_VIEW
- user_name: alice@example.com
level: CAN_MANAGEOther fields
其他字段
| Field | Use |
|---|---|
| Wall-clock limit per attempt, in minutes. |
| YAML equivalent of |
| Name the MLflow run, or choose the workspace directory that holds the experiment. |
| Run on a pre-provisioned GPU pool (find IDs with |
| Run different images and commands on different node ranks, for example a trainer and an inference server. Replaces |
| 字段 | 用途 |
|---|---|
| 每次尝试的挂钟时间限制,单位为分钟。 |
| |
| 为 MLflow 运行命名,或选择存放实验的工作区目录。 |
| 在预配的 GPU 池上运行(使用 |
| 为不同节点 rank 运行不同的镜像和命令,例如训练器和推理服务器。它会替代 |
Environment variables provided on each node
每个节点提供的环境变量
databricks aircommand:| Variable | Meaning |
|---|---|
| Absolute path to your extracted |
| Address/port of the rank-0 node for distributed coordination. |
| Total number of nodes. |
| Rank of the current node (0 to |
| Total number of GPUs across all nodes. |
| Number of GPUs on this node. |
| MLflow run ID for this job (log metrics/artifacts to it). |
| Path to the |
Your runs once per node, so per-process ranks (,
) are not set by the platform. Your distributed launcher
(torchrun, accelerate) assigns them when it spawns one process per GPU; pass it
/ / / .
command:RANKLOCAL_RANK$NODE_RANK$NUM_NODES$MASTER_ADDR$MASTER_PORTdatabricks aircommand:| 变量 | 含义 |
|---|---|
| 节点上解压后的 |
| 用于分布式协调的 rank-0 节点地址和端口。 |
| 节点总数。 |
| 当前节点的 rank(从 0 到 |
| 所有节点上的 GPU 总数。 |
| 当前节点上的 GPU 数量。 |
| 此作业的 MLflow 运行 ID(用于记录指标和产物)。 |
| |
你的 会在每个节点运行一次,因此平台不会设置进程级 rank(、)。分布式启动器(torchrun、accelerate)在每块 GPU 上启动一个进程时会分配这些变量;请向它传递 / / / 。
command:RANKLOCAL_RANK$NODE_RANK$NUM_NODES$MASTER_ADDR$MASTER_PORTStarting a run
启动运行
Start from a cookbook recipe when one fits. The
AI Runtime Cookbook
has runnable recipes, each a plus source code, grouped by domain
(, , ,
, , , and more; Ray recipes live in these
domain folders with a prefix). is the
minimal first run. Adapting the closest recipe is usually faster and less
error-prone than writing a workload from scratch:
workload.yamllanguage-models/computer-vision/recommendation-and-ranking/time-series/multimodal/serving/ray-getting-started/air-cli/bash
git clone https://github.com/databricks-solutions/ai-runtime-cookbook.git
cd ai-runtime-cookbook/<domain>/<recipe> # e.g. getting-started/air-cli适用时,请从 cookbook recipe 开始。
AI Runtime Cookbook
提供了可运行的配方,每个配方包含一个 和源代码,并按领域分组
(、、、
、、 等;Ray 配方位于这些领域目录中,并带有 前缀)。 是最小化的首次运行示例。适配最接近的配方通常比从零编写工作负载更快,也更不容易出错:
workload.yamllanguage-models/computer-vision/recommendation-and-ranking/time-series/multimodal/serving/ray-getting-started/air-cli/bash
git clone https://github.com/databricks-solutions/ai-runtime-cookbook.git
cd ai-runtime-cookbook/<domain>/<recipe> # e.g. getting-started/air-cliReview workload.yaml (and the recipe README) for values to configure, then:
Review workload.yaml (and the recipe README) for values to configure, then:
databricks air run --file workload.yaml --dry-run -o json -p <profile>
databricks air run --file workload.yaml --watch
--idempotency-key <key> -o json -p <profile> # key as in step 5
--idempotency-key <key> -o json -p <profile> # key as in step 5
Apply the steps below to the recipe's YAML the same way as to a new one,
including the stable `--idempotency-key` from step 5.
1. **Turn the user's intent into the YAML.** Extract model, dataset path, GPU
count and type, and the training command from the request. For gated models,
lead with the [token checklist](#hugging-face-gated-models) first. Ask before
synthesizing a config when a remaining critical detail is ambiguous (data
path, checkpoint dir, custom image).
2. **Write the `command:`** using the env vars above. Common launch shapes:
| Framework | `command:` shape |
|---|---|
| plain torch, single GPU | `python train.py` |
| torch, multi-GPU (1 node) | `torchrun --nproc_per_node=$LOCAL_WORLD_SIZE train.py` |
| torch, multi-node | `torchrun --nnodes=$NUM_NODES --nproc_per_node=$LOCAL_WORLD_SIZE --node_rank=$NODE_RANK --master_addr=$MASTER_ADDR --master_port=$MASTER_PORT train.py` |
| HuggingFace Accelerate | `accelerate launch --config_file accelerate.yaml train.py` |
| DeepSpeed | `deepspeed --num_gpus=$LOCAL_WORLD_SIZE train.py` |
| Composer / MosaicML | `composer train.py` |
Multi-line commands go under `command: |-`. Access uploaded code via
`$CODE_SOURCE_PATH`. **Total GPUs = nodes x accelerators-per-node**; see
[Accelerator types](#accelerator-types).
3. **Pre-flight the referenced resources** so the job does not die on startup:
- Every package the script imports is in the chosen
[environment](#python-environment) or its `dependencies` (the Standard
environment has no `torch`).
- `max_retries: 0` is set while the workload is new or being debugged.
- For each `secrets` reference, verify only its key name in the user's chosen
scope, filtering `list-secrets` as in the [token checklist](#hugging-face-gated-models).
Never browse other users' scopes or list their keys.
- If `environment.unity_catalog_image` is set, the image has been pushed to
Databricks Artifact Registry in the same workspace and the submitting
identity can read it (see [docker-images.md](docker-images.md)).
4. **Dry-run before submitting.** This validates the config without launching:
```bash
databricks air run --file workload.yaml --dry-run -o json -p <profile>On CLI 1.20.0 and newer, the dry run also checks the config against the
workspace, so it needs a working profile. A config-validation error is a
hard stop: it prints as plain text on stderr with a non-zero exit, so show
that message to the user and fix the YAML. Other failures are infra/auth
and should bubble up.
- Submit. Use a stable so an accidental re-run does not double-submit (compose it from
--idempotency-keyand bump<user>-<experiment>-<YYYYMMDD>-v<N>on each resubmit; avoid dynamic subexpressions likevNso the command stays auto-approvable):$(uuidgen)Addbashdatabricks air run --file workload.yaml \ --idempotency-key <key> -o json -p <profile>to stream logs to your terminal until the run finishes. Capture--watchfrom the response, then monitor it.data.run_id
databricks air run --file workload.yaml --dry-run -o json -p <profile>
databricks air run --file workload.yaml --watch
--idempotency-key <key> -o json -p <profile> # key as in step 5
--idempotency-key <key> -o json -p <profile> # key as in step 5
将以下步骤同样应用于配方的 YAML 和新编写的工作负载,包括使用第 5 步中的稳定 `--idempotency-key`。
1. **将用户意图转化为 YAML。** 从请求中提取模型、数据集路径、GPU 数量和类型以及训练命令。对于受限模型,先执行 [token 清单](#hugging-face-gated-models)。如果仍有关键细节不明确(数据路径、checkpoint 目录、自定义镜像),请先询问用户,不要直接合成配置。
2. **编写 `command:`**,使用上述环境变量。常见启动形式:
| 框架 | `command:` 形式 |
|---|---|
| 普通 torch,单 GPU | `python train.py` |
| torch,多 GPU(1 个节点) | `torchrun --nproc_per_node=$LOCAL_WORLD_SIZE train.py` |
| torch,多节点 | `torchrun --nnodes=$NUM_NODES --nproc_per_node=$LOCAL_WORLD_SIZE --node_rank=$NODE_RANK --master_addr=$MASTER_ADDR --master_port=$MASTER_PORT train.py` |
| HuggingFace Accelerate | `accelerate launch --config_file accelerate.yaml train.py` |
| DeepSpeed | `deepspeed --num_gpus=$LOCAL_WORLD_SIZE train.py` |
| Composer / MosaicML | `composer train.py` |
多行命令放在 `command: |-` 下。通过 `$CODE_SOURCE_PATH` 访问上传的代码。**GPU 总数 = 节点数 x 每节点加速器数**;参见[加速器类型](#accelerator-types)。
3. **预先检查引用的资源**,避免作业启动后失败:
- 脚本导入的每个包都应存在于所选的 [environment](#python-environment) 或其 `dependencies` 中(Standard 环境不包含 `torch`)。
- 工作负载刚创建或正在调试时,应设置 `max_retries: 0`。
- 对每条 `secrets` 引用,应按照 [token 清单](#hugging-face-gated-models) 中的方式,仅验证用户所选 scope 内的键名。绝不要浏览其他用户的 scopes 或列出其 keys。
- 如果设置了 `environment.unity_catalog_image`,则该镜像必须已推送到同一工作区的 Databricks Artifact Registry,并且提交身份具有读取权限(参见 [docker-images.md](docker-images.md))。
4. **提交前执行 dry run。** 这会在不启动作业的情况下验证配置:
```bash
databricks air run --file workload.yaml --dry-run -o json -p <profile>在 CLI 1.20.0 及更高版本中,dry run 还会根据工作区检查配置,因此需要有效的 profile。配置验证错误是硬性停止:它会以纯文本形式输出到 stderr 并返回非零退出码,因此应向用户显示该消息并修复 YAML。其他失败属于基础设施或身份验证问题,应向上传递。
- 提交。 使用稳定的 ,避免意外重跑导致重复提交(可由
--idempotency-key组成,并在每次重新提交时递增<user>-<experiment>-<YYYYMMDD>-v<N>;避免使用vN等动态子表达式,使命令保持可自动批准):$(uuidgen)添加bashdatabricks air run --file workload.yaml \ --idempotency-key <key> -o json -p <profile>可将日志流式传输到终端,直至运行结束。从响应中捕获--watch,然后对其进行监控。data.run_id
Checking on a run
检查运行
bash
databricks air get <JOB_RUN_ID> -o json -p <profile> # status, config, timing, failure reason
databricks air list -o json -p <profile> # your active runs
databricks air list --all-status -o json -p <profile> # include finished runsair getPENDINGRUNNINGSUCCESSFAILEDair list--all-statusair getbash
databricks air get <JOB_RUN_ID> -o json -p <profile> # status, config, timing, failure reason
databricks air list -o json -p <profile> # your active runs
databricks air list --all-status -o json -p <profile> # include finished runs当运行等待 GPU 时, 会报告 ,随后报告 ,最后进入 或 等最终状态。 返回空结果表示你没有活动运行,并不表示运行已消失;请使用 或 查看已结束的运行。
air getPENDINGRUNNINGSUCCESSFAILEDair list--all-statusair getMonitoring a run
监控运行
Stream logs while it runs. The simplest path is to submit with ,
which streams the run's logs to your terminal until it reaches a terminal state:
--watchbash
databricks air run --file workload.yaml --watch -p <profile>For a run that is already going, stream its logs by ID:
bash
databricks air logs <JOB_RUN_ID> -p <profile>运行时流式传输日志。 最简单的方法是在提交时添加 ,它会将运行日志流式传输到终端,直至运行进入终止状态:
--watchbash
databricks air run --file workload.yaml --watch -p <profile>对于已经在运行的作业,可通过 ID 流式传输其日志:
bash
databricks air logs <JOB_RUN_ID> -p <profile>--node N (one node), --tail N (last N lines of a finished run),
--node N (one node), --tail N (last N lines of a finished run),
--minutes N (last N minutes), --retry N (one retry attempt, 0-indexed),
--minutes N (last N minutes), --retry N (one retry attempt, 0-indexed),
--download-to DIR (save all nodes' logs)
--download-to DIR (save all nodes' logs)
**Loss, GPU utilization, and metric history live in MLflow.** The CLI streams
logs and status, not training metrics. The submit/status response includes the
MLflow experiment and run URL; echo it so the user can open the run in the
Databricks UI for loss curves, GPU utilization, and system charts.
**Loss、GPU 利用率和指标历史记录位于 MLflow。** CLI 会流式传输日志和状态,但不会提供训练指标。提交和状态响应包含 MLflow experiment 和 run URL;将其回显出来,以便用户可以在 Databricks UI 中打开该运行,查看 loss 曲线、GPU 利用率和系统图表。If a run fails
运行失败时
Get the failure reason, then pull the logs and surface the actual error to the
user:
bash
databricks air get <JOB_RUN_ID> -o json -p <profile> # termination_reason
databricks air logs <JOB_RUN_ID> --tail 500 -p <profile> # latest attempt
databricks air logs <JOB_RUN_ID> --retry 0 --tail 500 -p <profile> # first attemptIf the run was retried, the latest attempt's logs may show a different symptom
than the first failure; check . On a multi-node run, check other
nodes with when node 0 looks healthy.
--retry 0--node NShow the failing traceback and let the user decide the fix. Before
resubmitting, set so a deterministic error fails once. For a
deeper, guided diagnosis of a training failure, use AI Runtime's own debugging
support rather than guessing at root cause here.
max_retries: 0获取失败原因,然后拉取日志并向用户呈现实际错误:
bash
databricks air get <JOB_RUN_ID> -o json -p <profile> # termination_reason
databricks air logs <JOB_RUN_ID> --tail 500 -p <profile> # latest attempt
databricks air logs <JOB_RUN_ID> --retry 0 --tail 500 -p <profile> # first attempt如果运行经过重试,最新尝试的日志可能会显示与首次失败不同的症状;请检查 。对于多节点运行,当节点 0 看起来正常时,请使用 检查其他节点。
--retry 0--node N显示失败的 traceback,并让用户决定如何修复。重新提交前,将 设为 0,使确定性错误只失败一次。若需对训练失败进行更深入的引导式诊断,请使用 AI Runtime 自带的调试支持,而不是在这里猜测根本原因。
max_retriesServing a trained model
为训练后的模型提供服务
A model trained with is served as a
custom model:
log it with MLflow, register it in Unity Catalog, and create a serving endpoint
on CPU or GPU compute.
databricks airTo serve it with the library versions you tested, without pinning or
CUDA for the serving container, use an
express deployment
(Public Preview). The endpoint then runs the environment of the notebook you
registered from. Express deployments require:
torch- Logging and registering from an AI Runtime notebook to serve on GPU, or from
a serverless notebook to serve on CPU, with and
mlflow>=3.12.databricks-sdk>=0.102.0 - Registering with
. Do not use
mlflow.register_model(..., env_pack="databricks_model_serving")onregistered_model_name=: it does not acceptlog_modeland registers a standard model.env_pack - An environment of at most 200 GB.
For LLMs, use
Custom LLM Serving,
which builds on express deployments and starts an OpenAI-compatible server such
as vLLM from the model's metadata.
entrypointGPU names map to different GPUs per cloud. For example,
is an L40S (48 GB) on AWS and an A100 (80 GB) on Azure. See the GPU
table on the Custom LLM Serving page.
workload_typeGPU_LARGE使用 训练的模型可作为
custom model 提供服务:
将其记录到 MLflow,在 Unity Catalog 中注册,并在 CPU 或 GPU compute 上创建 serving endpoint。
databricks air要在不固定 serving 容器中 或 CUDA 版本的情况下,使用已测试的库版本提供服务,请使用
express deployment
(Public Preview)。该 endpoint 随后会运行你注册来源 notebook 的环境。Express deployments 要求:
torch- 从 AI Runtime notebook 记录并注册以在 GPU 上提供服务,或从 serverless notebook 记录并注册以在 CPU 上提供服务,并使用 和
mlflow>=3.12。databricks-sdk>=0.102.0 - 使用 进行注册。不要在
mlflow.register_model(..., env_pack="databricks_model_serving")上使用log_model:它不接受registered_model_name=,并且会注册标准模型。env_pack - 环境大小不超过 200 GB。
对于 LLM,请使用
Custom LLM Serving,
它基于 express deployments 构建,可根据模型的 元数据启动 OpenAI 兼容服务器,例如 vLLM。
entrypointGPU 名称会映射到不同云上的不同 GPU。例如, 在 AWS 上对应 L40S(48 GB),在 Azure 上对应 A100(80 GB)。请参阅 Custom LLM Serving 页面上的 GPU 表格。
workload_typeGPU_LARGECustom Docker images
自定义 Docker 镜像
Use a custom Docker image instead of when your workload
needs specific system libraries, CUDA extensions (flash-attn, apex, custom
kernels), or dependencies that do not fit .
environment.versionenvironment.dependenciesRead docker-images.md for step-by-step guidance on:
- Using Databricks-provided base images, including CUDA 13 images for B300
- Dockerfile patterns
- Pre-build compatibility checklist (CUDA/driver, PyTorch, NCCL, EFA/RDMA)
- Pushing images with and running them with
databricks air images pushenvironment.unity_catalog_image - Troubleshooting runtime errors such as on FIPS hosts
ssl.SSLError
当工作负载需要特定的系统库、CUDA 扩展(flash-attn、apex、自定义 kernel),或无法通过 安装的依赖时,请使用自定义 Docker 镜像代替 。
environment.dependenciesenvironment.version有关以下内容的分步指南,请阅读 docker-images.md:
- 使用 Databricks 提供的基础镜像,包括适用于 B300 的 CUDA 13 镜像
- Dockerfile 模式
- 预构建兼容性检查清单(CUDA/driver、PyTorch、NCCL、EFA/RDMA)
- 使用 推送镜像,并通过
databricks air images push运行镜像environment.unity_catalog_image - 排查运行时错误,例如 FIPS 主机上的
ssl.SSLError