databricks-ai-runtime

Compare original and translation side by side

🇺🇸

Original

English
🇨🇳

Translation

Chinese

Databricks AI Runtime (
databricks air
)

Databricks AI Runtime (
databricks air
)

Databricks AI Runtime is a set of
databricks air
CLI commands for submitting GPU training workloads to Databricks serverless compute. It manages environment setup, distributed training configuration, and workload lifecycle, without requiring you to manage clusters or infrastructure.
This skill covers the three everyday flows: starting a run, checking on a run, and monitoring a run as it trains.
Databricks AI Runtime 是一组
databricks air
CLI 命令,用于将 GPU 训练工作负载提交到 Databricks serverless compute。它负责管理环境设置、分布式训练配置和工作负载生命周期,无需你管理集群或基础设施。
本 skill 涵盖三个日常流程:启动运行、检查运行以及在训练期间监控运行。

Reference discipline

参考原则

The CLI's built-in help is the source of truth for field names, accelerator types, defaults, and constraints. Prefer it at runtime over anything memorized. Look up a config field by passing its path to
-h
on the
run
command (the path is a separate argument):
bash
databricks air run -h config                              # list the YAML fields
databricks air run -h config.compute                      # one field
databricks air run -h config.compute.accelerator_type     # a nested field
databricks air <subcommand> --help
documents each command's flags. This skill covers workflow and the few things the help does not spell out.
CLI 的内置帮助信息是字段名、加速器类型、默认值和约束条件的事实来源。运行时,应优先使用它,而不是依赖记忆中的内容。 若要查找某个配置字段,请将它的路径作为单独参数传给
run
命令的
-h
选项:
bash
databricks air run -h config                              # list the YAML fields
databricks air run -h config.compute                      # one field
databricks air run -h config.compute.accelerator_type     # a nested field
databricks air <subcommand> --help
记录每个命令的标志。本 skill 涵盖工作流以及帮助信息中未明确说明的少数内容。

Session setup

会话设置

Check the CLI version, confirm the command is available, and list your Databricks config profiles:
bash
databricks version
databricks air --help
databricks auth profiles --skip-validate    # available Databricks config profiles
The examples use flattened
code_source
fields, available in Databricks CLI 1.20.0 and newer. On an older CLI, use the equivalent nested snapshot format shown under Workload YAML. CLI 1.20.0 and newer accept both formats. Do not mix nested and flattened fields in the same configuration.
Pass
-p <profile>
on every command. Add
-o json
for scripted/programmatic calls so you get a structured envelope instead of human text:
  • Success:
    {"v": 1, "ts": "...", "data": {...}}
  • Error:
    {"v": 1, "ts": "...", "error": {"code": "...", "kind": "...", "message": "...", "retryable": bool}}
Config validation (input) errors are the exception: they print as plain text to stderr with a non-zero exit even under
-o json
, so check the exit code and stderr too, not just the parsed JSON. When you do get a structured error, its
kind
tells you what to do:
kindMeaningAction
PERMANENT
Not authenticated / permission denied / feature disabled / non-retryableIf auth:
databricks auth login --host <workspace-url>
NOT_FOUND
Bad run ID / resourceCheck the input
CONFLICT
Idempotency key collisionUse a new
--idempotency-key
TRANSIENT
RetryableRetry
检查 CLI 版本,确认命令可用,并列出你的 Databricks 配置档案:
bash
databricks version
databricks air --help
databricks auth profiles --skip-validate    # available Databricks config profiles
示例使用扁平化的
code_source
字段,该形式适用于 Databricks CLI 1.20.0 及更高版本。 对于较旧的 CLI,请使用 Workload YAML 下所示的等效嵌套 snapshot 格式。Databricks CLI 1.20.0 及更高版本接受这两种格式。 不要在同一配置中混用嵌套字段和扁平化字段。
每个命令都传入
-p <profile>
。对于脚本化或编程式调用,添加
-o json
,这样获得的是结构化封装,而不是人类可读的文本:
  • 成功:
    {"v": 1, "ts": "...", "data": {...}}
  • 错误:
    {"v": 1, "ts": "...", "error": {"code": "...", "kind": "...", "message": "...", "retryable": bool}}
配置验证(输入)错误是例外:即使在
-o json
下,也会以纯文本形式输出到 stderr 并返回非零退出码,因此除了解析后的 JSON,还要检查退出码和 stderr。获得结构化错误时,其
kind
会告诉你应采取什么操作:
kind含义操作
PERMANENT
未认证、权限被拒、功能未启用或不可重试如果是身份验证问题:
databricks auth login --host <workspace-url>
NOT_FOUND
运行 ID 或资源无效检查输入
CONFLICT
Idempotency key 冲突使用新的
--idempotency-key
TRANSIENT
可重试重试

Subcommands

子命令

CommandWhat it does
databricks air run
Submit a workload from a YAML file. Add
--watch
to stream logs until it finishes,
--dry-run
to validate without submitting, and
--override <field>=<value>
(repeatable) to change a field without editing the YAML.
databricks air get <JOB_RUN_ID>
Show status, config, timing, and the failure reason for one run.
databricks air list
List your active runs. A run that just finished drops out of the default view; add
--all-status
to include finished runs. Also
--all-users
,
--filter KEY=VALUE
,
--limit
.
databricks air cancel <JOB_RUN_ID> [<JOB_RUN_ID>...]
/
... --all
Cancel one or more runs, or all of your active runs (
--all
prompts; add
--yes
in scripts).
databricks air logs <JOB_RUN_ID>
Stream a running run's logs, or fetch a finished run's logs.
databricks air pools list
/
pools get <POOL_ID>
View the provisioned GPU pools available to the workspace.
databricks air images push
Push a local or remote container image to Databricks Artifact Registry. See docker-images.md.
databricks air convert-to-dabs <yaml_path>
Convert a workload YAML into a Databricks Asset Bundle (local only; deploy with
databricks bundle deploy
).
For flags and defaults, run
databricks air <subcommand> --help
.
命令作用
databricks air run
从 YAML 文件提交工作负载。添加
--watch
可流式传输日志直至运行结束,添加
--dry-run
可在不提交的情况下验证配置,添加
--override <field>=<value>
(可重复使用)可在不编辑 YAML 的情况下修改字段。
databricks air get <JOB_RUN_ID>
显示某次运行的状态、配置、耗时和失败原因。
databricks air list
列出你的活动运行。刚完成的运行会从默认视图中消失;添加
--all-status
可包含已结束的运行。还支持
--all-users
、
--filter KEY=VALUE
和
--limit
。
databricks air cancel <JOB_RUN_ID> [<JOB_RUN_ID>...]
/
... --all
取消一个或多个运行,或取消你的所有活动运行(
--all
会提示确认;在脚本中添加
--yes
)。
databricks air logs <JOB_RUN_ID>
流式传输正在运行的日志,或获取已结束运行的日志。
databricks air pools list
/
pools get <POOL_ID>
查看工作区中可用的预配 GPU 池。
databricks air images push
将本地或远程容器镜像推送到 Databricks Artifact Registry。参见 docker-images.md。
databricks air convert-to-dabs <yaml_path>
将工作负载 YAML 转换为 Databricks Asset Bundle(仅在本地执行;使用
databricks bundle deploy
部署)。
有关标志和默认值,请运行
databricks air <subcommand> --help
。

Workload YAML

工作负载 YAML

yaml
experiment_name: my-training-job     # alphanumeric, - and _ only
compute:
  num_accelerators: 1
  accelerator_type: GPU_1xA10        # run `databricks air run -h config.compute` for valid values
environment:
  version: "6"                       # Standard environment; use databricks_ai_v6 for PyTorch preinstalled
  dependencies:
    - torch
    - mlflow
code_source:
  root_path: "."                     # relative to workload.yaml; extracted to $CODE_SOURCE_PATH on each node
max_retries: 0                       # while iterating; see Retries below
command: |-
  cd "$CODE_SOURCE_PATH"
  python train.py
Submit with
databricks air run --file workload.yaml -p <profile>
. Field details live in
databricks air run -h config.<field>
(the authoritative source).
For an older CLI, replace only the
code_source
block above with:
yaml
code_source:
  type: snapshot
  snapshot:
    root_path: "."
The values and behavior are the same: add
type: snapshot
under
code_source
and put all snapshot fields (
root_path
, and any
remote_volume
,
git
, or
include_paths
) inside
code_source.snapshot
. Keep the rest of the workload unchanged. Check
databricks air run -h config.code_source
for the fields supported by the installed CLI.
yaml
experiment_name: my-training-job     # alphanumeric, - and _ only
compute:
  num_accelerators: 1
  accelerator_type: GPU_1xA10        # run `databricks air run -h config.compute` for valid values
environment:
  version: "6"                       # Standard environment; use databricks_ai_v6 for PyTorch preinstalled
  dependencies:
    - torch
    - mlflow
code_source:
  root_path: "."                     # relative to workload.yaml; extracted to $CODE_SOURCE_PATH on each node
max_retries: 0                       # while iterating; see Retries below
command: |-
  cd "$CODE_SOURCE_PATH"
  python train.py
使用
databricks air run --file workload.yaml -p <profile>
提交。字段详情位于
databricks air run -h config.<field>
(权威来源)。
对于较旧的 CLI,只需将上面的
code_source
块替换为:
yaml
code_source:
  type: snapshot
  snapshot:
    root_path: "."
这些值和行为完全相同:在
code_source
下添加
type: snapshot
,并将所有 snapshot 字段(
root_path
,以及可选的
remote_volume
、
git
或
include_paths
)放在
code_source.snapshot
内。其余工作负载保持不变。运行
databricks air run -h config.code_source
,查看已安装 CLI 支持的字段。

Accelerator types

加速器类型

Run
databricks air run -h config.compute
to see the accelerator types. On CLI 1.23.0 and newer it lists the types available in the current workspace and how many GPUs each type has per node; if it cannot reach the workspace, it prints a warning and falls back to a built-in list. Do not guess a type from memory: availability depends on the workspace, cloud, and region, and matching is case-sensitive.
Type names follow
GPU_<GPUs per node>x<GPU>
, for example
GPU_1xA10
,
GPU_1xH100
,
GPU_8xH100
, and
GPU_8xB300
.
num_accelerators
is the total GPU count across all nodes, so it must be a multiple of the per-node count: 8 on an
8x
type is one node, 16 is two nodes. Only the
8x
types support multi-GPU and multi-node training.
运行
databricks air run -h config.compute
查看加速器类型。在 CLI 1.23.0 及更高版本中,它会列出当前工作区中可用的类型以及每种类型在每个节点上的 GPU 数量;如果无法连接工作区,则会打印警告并回退到内置列表。不要凭记忆猜测类型:可用性取决于工作区、云和区域,且匹配区分大小写。
类型名称遵循
GPU_<每节点 GPU 数量>x<GPU>
,例如
GPU_1xA10
、
GPU_1xH100
、
GPU_8xH100
和
GPU_8xB300
。
num_accelerators
是所有节点的 GPU 总数,因此它必须是每节点数量的倍数:在
8x
类型上,8 表示一个节点,16 表示两个节点。只有
8x
类型支持多 GPU 和多节点训练。

Python environment

Python 环境

Pick the environment from what the training code imports, before you submit, so a missing package does not surface only after a GPU has been provisioned:
environment
What it providesUse when
version: "6"
(the default)
Standard environment: Python and CUDA only. Since version 5, it does not include
torch
or
torchvision
.
You control the dependency stack. List every import, including
torch
, under
dependencies
.
version: databricks_ai_v6
Databricks AI environment: PyTorch, Transformers, and other ML frameworks preinstalled.Typical PyTorch or Hugging Face training. Add extras under
dependencies
.
unity_catalog_image: <catalog>.<schema>.<image>:<tag>
Your own Docker image.System libraries or CUDA extensions; see Custom Docker images.
Before submitting, check the script's imports against the chosen environment. For the exact packages in each environment version, see the environment version release notes. Do not combine
unity_catalog_image
with
version
or
dependencies
.
提交前,应根据训练代码导入的内容选择环境,避免等到 GPU 已预配后才发现缺少包:
environment
提供的内容使用场景
version: "6"
(默认值)
标准环境:仅包含 Python 和 CUDA。从版本 5 开始,它不包含
torch
或
torchvision
。
你自行控制依赖栈。将所有 import,包括
torch
,列在
dependencies
中。
version: databricks_ai_v6
Databricks AI 环境:预装 PyTorch、Transformers 和其他 ML 框架。典型的 PyTorch 或 Hugging Face 训练。在
dependencies
中添加额外依赖。
unity_catalog_image: <catalog>.<schema>.<image>:<tag>
你自己的 Docker 镜像。需要系统库或 CUDA 扩展;参见自定义 Docker 镜像。
提交前,将脚本中的 import 与所选环境进行核对。每个环境版本中的确切软件包,请参阅环境版本发布说明。 不要将
unity_catalog_image
与
version
或
dependencies
结合使用。

Configuring the workload

配置工作负载

These optional top-level fields cover the common needs. Run
databricks air run -h config.<field>
for the full reference on any of them.
以下可选的顶层字段涵盖了常见需求。对于其中任何一个字段,运行
databricks air run -h config.<field>
即可获取完整参考。

Hugging Face gated models

Hugging Face 受限模型

When a model needs a Hugging Face (HF) token, lead with this checklist before asking for other training details or running secrets commands:
  1. Accept the model license on HF and, if required, wait for access approval. Create a fine-grained read-only token with Read access to contents of all public gated repos you can access. For a private repo, also grant read access to that specific repo.
  2. Choose an existing scope the user owns and can write. If they need to find one, have them filter
    list-scopes
    to their own username or scopes they created, then confirm ownership and access; a name match alone is not proof. For example, the user can run the following with their own username:
    bash
    databricks secrets list-scopes -p <profile> -o json | jq -r --arg user "<your-username>" '.[] | select(.name | contains($user)) | .name'
    Never browse or inspect other users' scopes or keys. Create a scope only if needed; if
    create-scope
    reports
    Max limit of 2000 scopes has been reached
    , reuse an existing user-owned scope instead. The user may need to run
    list-acls
    on their own scope to check access. Before using
    <scope>
    in any
    databricks secrets ...
    command, validate it: it must be an existing scope the user owns, from their own scopes, and match the scope-name charset
    ^[A-Za-z0-9_.@-]+$
    . If it doesn't, have the user run the commands themselves. Always double-quote it. If the scope is Azure Key Vault-backed (
    backend_type
    is
    AZURE_KEYVAULT
    ),
    put-secret
    can't write to it: have the user add the token to that vault with their own tooling and use that secret's name as the key below.
  3. First check whether
    hf-token
    already exists in the scope, using the key-name check in step 4. If it does, reuse it only if it is the right token; otherwise pick a new key such as
    hf-token-gemma
    , or get the user's explicit OK before overwriting it, since other workloads may read it. Use the chosen key in steps 4 and 5. Have the user run this in their own terminal and enter the token at the hidden prompt. Never paste the token into chat or a command line.
    bash
    databricks secrets put-secret "<scope>" hf-token -p <profile>
  4. Once they have stored it and named their scope, verify only the key name in that one scope, filtering the output to
    hf-token
    :
    bash
    databricks secrets list-secrets "<scope>" -p <profile> -o json | jq -r '.[] | select(.key == "hf-token") | .key'
  5. Reference
    secrets: {HF_TOKEN: "<scope>/hf-token"}
    in the workload. Quote the reference, particularly when the scope contains
    @
    or
    .
    . Transformers reads
    HF_TOKEN
    automatically.
Gemma 3 is gated; Mistral Small 3.2, gpt-oss-20b, and Qwen3 are ungated and do not need a token just to download their public weights. Check the specific model's HF page for current access requirements.
当模型需要 Hugging Face (HF) token 时,在询问其他训练详情或运行 secrets 命令之前,先按照以下清单操作:
  1. 在 HF 上接受模型许可证;如需审批,则等待访问权限获批。创建一个 fine-grained 只读 token,并授予 Read access to contents of all public gated repos you can access。对于私有仓库,还需授予对该特定仓库的读取权限。
  2. 选择一个用户拥有且可写的现有 scope。如果用户需要查找,可让用户运行
    list-scopes
    ,按自己的用户名或创建的 scopes 进行筛选,然后确认所有权和访问权限;仅名称匹配并不能证明所有权。例如,用户可以使用自己的用户名运行以下命令:
    bash
    databricks secrets list-scopes -p <profile> -o json | jq -r --arg user "<your-username>" '.[] | select(.name | contains($user)) | .name'
    绝不要浏览或检查其他用户的 scopes 或 keys。仅在确有需要时创建 scope;如果
    create-scope
    报告
    Max limit of 2000 scopes has been reached
    ,则改为复用现有的用户自有 scope。用户可能需要在自己的 scope 上运行
    list-acls
    以检查访问权限。在任何
    databricks secrets ...
    命令中使用
    <scope>
    前,先进行验证:它必须是用户从自己的 scopes 中选取且拥有的现有 scope,并符合 scope 名称字符集
    ^[A-Za-z0-9_.@-]+$
    。如果不符,应让用户自行运行这些命令。始终使用双引号包裹它。如果该 scope 由 Azure Key Vault 支撑(
    backend_type
    为
    AZURE_KEYVAULT
    ),
    put-secret
    将无法向其写入:应让用户使用自己的工具将 token 添加到该 vault,并把该 secret 的名称用作下面的 key。
  3. 使用第 4 步的键名检查方式,先检查该 scope 中是否已存在
    hf-token
    。如果存在,仅在它确实是所需 token 时复用;否则请选择新 key,例如
    hf-token-gemma
    ,或者在覆盖前获得用户的明确同意,因为其他工作负载可能读取该 key。在第 4 步和第 5 步中使用选定的 key。让用户在自己的终端中运行以下命令,并在隐藏输入提示中输入 token。绝不要将 token 粘贴到聊天或命令行中。
    bash
    databricks secrets put-secret "<scope>" hf-token -p <profile>
  4. 用户存储 token 并提供 scope 名称后,仅验证该 scope 中的键名,并将输出筛选为
    hf-token
    :
    bash
    databricks secrets list-secrets "<scope>" -p <profile> -o json | jq -r '.[] | select(.key == "hf-token") | .key'
  5. 在工作负载中引用
    secrets: {HF_TOKEN: "<scope>/hf-token"}
    。请为引用添加引号,尤其是在 scope 包含
    @
    或
    .
    时。Transformers 会自动读取
    HF_TOKEN
    。
Gemma 3 属于受限模型;Mistral Small 3.2、gpt-oss-20b 和 Qwen3 无访问限制,仅下载公开权重不需要 token。请查看特定模型的 HF 页面以了解当前的访问要求。

Environment variables and secrets

环境变量和密钥

Set plain values under
env_variables
; pull sensitive values from Databricks Secrets under
secrets
(each entry maps an env var name to a
scope/key
, and the value is materialized on the worker, never logged):
yaml
env_variables:
  BATCH_SIZE: "32"
secrets:
  HF_TOKEN: "<scope>/hf-token"          # -> $HF_TOKEN on every node
  WANDB_API_KEY: "<scope>/wandb_key"
The scope and keys must already exist. Prefer a user-owned scope; create one only if needed, and have the user store secret values in their own terminal. Secret env var names must not collide with
env_variables
names.
在
env_variables
下设置普通值;从 Databricks Secrets 中获取敏感值并放入
secrets
(每个条目将环境变量名映射到
scope/key
,值会在 worker 上实例化,且永不写入日志):
yaml
env_variables:
  BATCH_SIZE: "32"
secrets:
  HF_TOKEN: "<scope>/hf-token"          # -> $HF_TOKEN on every node
  WANDB_API_KEY: "<scope>/wandb_key"
scope 和 key 必须已经存在。优先使用用户自有 scope;仅在需要时创建,并让用户在自己的终端中存储 secret 值。Secret 环境变量名不得与
env_variables
中的名称冲突。

Parameters (hyperparameters)

参数(超参数)

parameters
is the clean way to pass hyperparameters into your script and change them between runs without editing training code. The CLI writes them to a YAML file on the cluster and exposes its path as
$HYPERPARAMETERS_PATH
:
yaml
parameters:
  learning_rate: 0.001
  batch_size: 32
  warmup_steps: 1000
  use_amp: true
Read them in the script:
python
import os, yaml
with open(os.environ["HYPERPARAMETERS_PATH"]) as f:
    hp = yaml.safe_load(f)
lr = hp["learning_rate"]
Values can be strings, numbers, booleans, lists, or nested dicts. To sweep a hyperparameter without editing the YAML, resubmit with
--override parameters.learning_rate=0.0005
(and a fresh idempotency key).
parameters
是一种简洁方式,可将超参数传入脚本,并在不同运行之间修改超参数,而无需编辑训练代码。CLI 会将其写入集群上的 YAML 文件,并将其路径公开为
$HYPERPARAMETERS_PATH
:
yaml
parameters:
  learning_rate: 0.001
  batch_size: 32
  warmup_steps: 1000
  use_amp: true
在脚本中读取它们:
python
import os, yaml
with open(os.environ["HYPERPARAMETERS_PATH"]) as f:
    hp = yaml.safe_load(f)
lr = hp["learning_rate"]
值可以是字符串、数字、布尔值、列表或嵌套字典。要在不编辑 YAML 的情况下扫描某个超参数,请使用
--override parameters.learning_rate=0.0005
重新提交(并使用新的 idempotency key)。

Retries

重试

Failed attempts are retried automatically:
max_retries
defaults to 3 when unset (confirm with
databricks air run -h config.max_retries
). Each retry provisions GPUs and reruns the whole command, so a deterministic failure (an import error, a typo, a bad path) fails 4 times before the run gives up.
Set
max_retries: 0
while writing, debugging, or iterating on a workload
, either in the YAML or with
--override max_retries=0
. Raise it only for a long, stable training run where retrying can recover from transient infrastructure failures, and make the training code resume from a checkpoint so a retry does not start over. When
timeout_minutes
is set, it applies per attempt.
失败的尝试会自动重试:未设置时,
max_retries
默认为 3(通过
databricks air run -h config.max_retries
确认)。每次重试都会重新预配 GPU 并重新运行整个命令,因此确定性失败(import 错误、拼写错误、错误路径)会在任务放弃前失败 4 次。
编写、调试或迭代工作负载时,将
max_retries
设为 0
,无论是在 YAML 中设置,还是使用
--override max_retries=0
。仅当长时间且稳定的训练任务可以通过重试从暂时性基础设施故障中恢复时,才提高该值;同时应确保训练代码可从 checkpoint 恢复,避免重试时从头开始。设置
timeout_minutes
时,它适用于每次尝试。

Permissions

权限

By default only you can see the run. Grant others access to the submitted job with
permissions
(DABs-compatible format); levels are
CAN_VIEW
,
CAN_MANAGE_RUN
,
CAN_MANAGE
:
yaml
permissions:
  - group_name: data-engineering
    level: CAN_VIEW
  - user_name: alice@example.com
    level: CAN_MANAGE
默认情况下,只有你能看到该运行。使用
permissions
(兼容 DABs 的格式)授予他人访问所提交作业的权限;权限级别为
CAN_VIEW
、
CAN_MANAGE_RUN
、
CAN_MANAGE
:
yaml
permissions:
  - group_name: data-engineering
    level: CAN_VIEW
  - user_name: alice@example.com
    level: CAN_MANAGE

Other fields

其他字段

FieldUse
timeout_minutes
Wall-clock limit per attempt, in minutes.
idempotency_token
YAML equivalent of
--idempotency-key
: resubmitting with the same token returns the existing run.
mlflow_run_name
,
mlflow_experiment_directory
Name the MLflow run, or choose the workspace directory that holds the experiment.
compute.pool_id
,
compute.priority_class
Run on a pre-provisioned GPU pool (find IDs with
databricks air pools list
).
priority_class
is
NORMAL
or
BEST_EFFORT
(preemptable) and requires
pool_id
.
containers
Run different images and commands on different node ranks, for example a trainer and an inference server. Replaces
command
; each entry needs
name
,
command
,
ranks
, and
unity_catalog_image
. See
databricks air run -h config.containers
.
字段用途
timeout_minutes
每次尝试的挂钟时间限制,单位为分钟。
idempotency_token
--idempotency-key
的 YAML 等效项:使用相同 token 重新提交会返回现有运行。
mlflow_run_name
、
mlflow_experiment_directory
为 MLflow 运行命名,或选择存放实验的工作区目录。
compute.pool_id
、
compute.priority_class
在预配的 GPU 池上运行(使用
databricks air pools list
查找 ID)。
priority_class
为
NORMAL
或
BEST_EFFORT
(可抢占),且必须设置
pool_id
。
containers
为不同节点 rank 运行不同的镜像和命令,例如训练器和推理服务器。它会替代
command
;每个条目都需要
name
、
command
、
ranks
和
unity_catalog_image
。参见
databricks air run -h config.containers
。

Environment variables provided on each node

每个节点提供的环境变量

databricks air
sets these automatically on every node; use them in
command:
and in your training code.
VariableMeaning
CODE_SOURCE_PATH
Absolute path to your extracted
code_source
on the node (
cd $CODE_SOURCE_PATH
).
MASTER_ADDR
/
MASTER_PORT
Address/port of the rank-0 node for distributed coordination.
NUM_NODES
Total number of nodes.
NODE_RANK
Rank of the current node (0 to
NUM_NODES
-1). Also set as
POD_RANK
.
WORLD_SIZE
Total number of GPUs across all nodes.
LOCAL_WORLD_SIZE
Number of GPUs on this node.
MLFLOW_RUN_ID
MLflow run ID for this job (log metrics/artifacts to it).
HYPERPARAMETERS_PATH
Path to the
parameters
YAML file (only set when
parameters:
is defined).
Your
command:
runs once per node, so per-process ranks (
RANK
,
LOCAL_RANK
) are not set by the platform. Your distributed launcher (torchrun, accelerate) assigns them when it spawns one process per GPU; pass it
$NODE_RANK
/
$NUM_NODES
/
$MASTER_ADDR
/
$MASTER_PORT
.
databricks air
会在每个节点自动设置以下变量;可在
command:
和训练代码中使用。
变量含义
CODE_SOURCE_PATH
节点上解压后的
code_source
绝对路径(
cd $CODE_SOURCE_PATH
)。
MASTER_ADDR
/
MASTER_PORT
用于分布式协调的 rank-0 节点地址和端口。
NUM_NODES
节点总数。
NODE_RANK
当前节点的 rank(从 0 到
NUM_NODES
-1)。同时也会设置为
POD_RANK
。
WORLD_SIZE
所有节点上的 GPU 总数。
LOCAL_WORLD_SIZE
当前节点上的 GPU 数量。
MLFLOW_RUN_ID
此作业的 MLflow 运行 ID(用于记录指标和产物)。
HYPERPARAMETERS_PATH
parameters
YAML 文件的路径(仅在定义了
parameters:
时设置)。
你的
command:
会在每个节点运行一次,因此平台不会设置进程级 rank(
RANK
、
LOCAL_RANK
)。分布式启动器(torchrun、accelerate)在每块 GPU 上启动一个进程时会分配这些变量;请向它传递
$NODE_RANK
/
$NUM_NODES
/
$MASTER_ADDR
/
$MASTER_PORT
。

Starting a run

启动运行

Start from a cookbook recipe when one fits. The AI Runtime Cookbook has runnable recipes, each a
workload.yaml
plus source code, grouped by domain (
language-models/
,
computer-vision/
,
recommendation-and-ranking/
,
time-series/
,
multimodal/
,
serving/
, and more; Ray recipes live in these domain folders with a
ray-
prefix).
getting-started/air-cli/
is the minimal first run. Adapting the closest recipe is usually faster and less error-prone than writing a workload from scratch:
bash
git clone https://github.com/databricks-solutions/ai-runtime-cookbook.git
cd ai-runtime-cookbook/<domain>/<recipe>     # e.g. getting-started/air-cli
适用时,请从 cookbook recipe 开始。 AI Runtime Cookbook 提供了可运行的配方,每个配方包含一个
workload.yaml
和源代码,并按领域分组 (
language-models/
、
computer-vision/
、
recommendation-and-ranking/
、
time-series/
、
multimodal/
、
serving/
等;Ray 配方位于这些领域目录中,并带有
ray-
前缀)。
getting-started/air-cli/
是最小化的首次运行示例。适配最接近的配方通常比从零编写工作负载更快,也更不容易出错:
bash
git clone https://github.com/databricks-solutions/ai-runtime-cookbook.git
cd ai-runtime-cookbook/<domain>/<recipe>     # e.g. getting-started/air-cli

Review workload.yaml (and the recipe README) for values to configure, then:

Review workload.yaml (and the recipe README) for values to configure, then:

databricks air run --file workload.yaml --dry-run -o json -p <profile> databricks air run --file workload.yaml --watch
--idempotency-key <key> -o json -p <profile> # key as in step 5

Apply the steps below to the recipe's YAML the same way as to a new one,
including the stable `--idempotency-key` from step 5.

1. **Turn the user's intent into the YAML.** Extract model, dataset path, GPU
   count and type, and the training command from the request. For gated models,
   lead with the [token checklist](#hugging-face-gated-models) first. Ask before
   synthesizing a config when a remaining critical detail is ambiguous (data
   path, checkpoint dir, custom image).

2. **Write the `command:`** using the env vars above. Common launch shapes:

   | Framework | `command:` shape |
   |---|---|
   | plain torch, single GPU | `python train.py` |
   | torch, multi-GPU (1 node) | `torchrun --nproc_per_node=$LOCAL_WORLD_SIZE train.py` |
   | torch, multi-node | `torchrun --nnodes=$NUM_NODES --nproc_per_node=$LOCAL_WORLD_SIZE --node_rank=$NODE_RANK --master_addr=$MASTER_ADDR --master_port=$MASTER_PORT train.py` |
   | HuggingFace Accelerate | `accelerate launch --config_file accelerate.yaml train.py` |
   | DeepSpeed | `deepspeed --num_gpus=$LOCAL_WORLD_SIZE train.py` |
   | Composer / MosaicML | `composer train.py` |

   Multi-line commands go under `command: |-`. Access uploaded code via
   `$CODE_SOURCE_PATH`. **Total GPUs = nodes x accelerators-per-node**; see
   [Accelerator types](#accelerator-types).

3. **Pre-flight the referenced resources** so the job does not die on startup:
   - Every package the script imports is in the chosen
     [environment](#python-environment) or its `dependencies` (the Standard
     environment has no `torch`).
   - `max_retries: 0` is set while the workload is new or being debugged.
   - For each `secrets` reference, verify only its key name in the user's chosen
     scope, filtering `list-secrets` as in the [token checklist](#hugging-face-gated-models).
     Never browse other users' scopes or list their keys.
   - If `environment.unity_catalog_image` is set, the image has been pushed to
     Databricks Artifact Registry in the same workspace and the submitting
     identity can read it (see [docker-images.md](docker-images.md)).

4. **Dry-run before submitting.** This validates the config without launching:
   ```bash
   databricks air run --file workload.yaml --dry-run -o json -p <profile>
On CLI 1.20.0 and newer, the dry run also checks the config against the workspace, so it needs a working profile. A config-validation error is a hard stop: it prints as plain text on stderr with a non-zero exit, so show that message to the user and fix the YAML. Other failures are infra/auth and should bubble up.
  1. Submit. Use a stable
    --idempotency-key
    so an accidental re-run does not double-submit (compose it from
    <user>-<experiment>-<YYYYMMDD>-v<N>
    and bump
    vN
    on each resubmit; avoid dynamic subexpressions like
    $(uuidgen)
    so the command stays auto-approvable):
    bash
    databricks air run --file workload.yaml \
      --idempotency-key <key> -o json -p <profile>
    Add
    --watch
    to stream logs to your terminal until the run finishes. Capture
    data.run_id
    from the response, then monitor it.
databricks air run --file workload.yaml --dry-run -o json -p <profile> databricks air run --file workload.yaml --watch
--idempotency-key <key> -o json -p <profile> # key as in step 5

将以下步骤同样应用于配方的 YAML 和新编写的工作负载,包括使用第 5 步中的稳定 `--idempotency-key`。

1. **将用户意图转化为 YAML。** 从请求中提取模型、数据集路径、GPU 数量和类型以及训练命令。对于受限模型,先执行 [token 清单](#hugging-face-gated-models)。如果仍有关键细节不明确(数据路径、checkpoint 目录、自定义镜像),请先询问用户,不要直接合成配置。

2. **编写 `command:`**,使用上述环境变量。常见启动形式:

   | 框架 | `command:` 形式 |
   |---|---|
   | 普通 torch,单 GPU | `python train.py` |
   | torch,多 GPU(1 个节点) | `torchrun --nproc_per_node=$LOCAL_WORLD_SIZE train.py` |
   | torch,多节点 | `torchrun --nnodes=$NUM_NODES --nproc_per_node=$LOCAL_WORLD_SIZE --node_rank=$NODE_RANK --master_addr=$MASTER_ADDR --master_port=$MASTER_PORT train.py` |
   | HuggingFace Accelerate | `accelerate launch --config_file accelerate.yaml train.py` |
   | DeepSpeed | `deepspeed --num_gpus=$LOCAL_WORLD_SIZE train.py` |
   | Composer / MosaicML | `composer train.py` |

   多行命令放在 `command: |-` 下。通过 `$CODE_SOURCE_PATH` 访问上传的代码。**GPU 总数 = 节点数 x 每节点加速器数**;参见[加速器类型](#accelerator-types)。

3. **预先检查引用的资源**,避免作业启动后失败:
   - 脚本导入的每个包都应存在于所选的 [environment](#python-environment) 或其 `dependencies` 中(Standard 环境不包含 `torch`)。
   - 工作负载刚创建或正在调试时,应设置 `max_retries: 0`。
   - 对每条 `secrets` 引用,应按照 [token 清单](#hugging-face-gated-models) 中的方式,仅验证用户所选 scope 内的键名。绝不要浏览其他用户的 scopes 或列出其 keys。
   - 如果设置了 `environment.unity_catalog_image`,则该镜像必须已推送到同一工作区的 Databricks Artifact Registry,并且提交身份具有读取权限(参见 [docker-images.md](docker-images.md))。

4. **提交前执行 dry run。** 这会在不启动作业的情况下验证配置:
   ```bash
   databricks air run --file workload.yaml --dry-run -o json -p <profile>
在 CLI 1.20.0 及更高版本中,dry run 还会根据工作区检查配置,因此需要有效的 profile。配置验证错误是硬性停止:它会以纯文本形式输出到 stderr 并返回非零退出码,因此应向用户显示该消息并修复 YAML。其他失败属于基础设施或身份验证问题,应向上传递。
  1. 提交。 使用稳定的
    --idempotency-key
    ,避免意外重跑导致重复提交(可由
    <user>-<experiment>-<YYYYMMDD>-v<N>
    组成,并在每次重新提交时递增
    vN
    ;避免使用
    $(uuidgen)
    等动态子表达式,使命令保持可自动批准):
    bash
    databricks air run --file workload.yaml \
      --idempotency-key <key> -o json -p <profile>
    添加
    --watch
    可将日志流式传输到终端,直至运行结束。从响应中捕获
    data.run_id
    ,然后对其进行监控。

Checking on a run

检查运行

bash
databricks air get <JOB_RUN_ID> -o json -p <profile>        # status, config, timing, failure reason
databricks air list -o json -p <profile>                    # your active runs
databricks air list --all-status -o json -p <profile>       # include finished runs
air get
reports
PENDING
while the run waits for GPUs, then
RUNNING
, then a final state such as
SUCCESS
or
FAILED
. An empty
air list
means you have no active runs, not that the run is gone; use
--all-status
or
air get
to see a finished run.
bash
databricks air get <JOB_RUN_ID> -o json -p <profile>        # status, config, timing, failure reason
databricks air list -o json -p <profile>                    # your active runs
databricks air list --all-status -o json -p <profile>       # include finished runs
当运行等待 GPU 时,
air get
会报告
PENDING
,随后报告
RUNNING
,最后进入
SUCCESS
或
FAILED
等最终状态。
air list
返回空结果表示你没有活动运行,并不表示运行已消失;请使用
--all-status
或
air get
查看已结束的运行。

Monitoring a run

监控运行

Stream logs while it runs. The simplest path is to submit with
--watch
, which streams the run's logs to your terminal until it reaches a terminal state:
bash
databricks air run --file workload.yaml --watch -p <profile>
For a run that is already going, stream its logs by ID:
bash
databricks air logs <JOB_RUN_ID> -p <profile>
运行时流式传输日志。 最简单的方法是在提交时添加
--watch
,它会将运行日志流式传输到终端,直至运行进入终止状态:
bash
databricks air run --file workload.yaml --watch -p <profile>
对于已经在运行的作业,可通过 ID 流式传输其日志:
bash
databricks air logs <JOB_RUN_ID> -p <profile>

--node N (one node), --tail N (last N lines of a finished run),

--node N (one node), --tail N (last N lines of a finished run),

--minutes N (last N minutes), --retry N (one retry attempt, 0-indexed),

--minutes N (last N minutes), --retry N (one retry attempt, 0-indexed),

--download-to DIR (save all nodes' logs)

--download-to DIR (save all nodes' logs)


**Loss, GPU utilization, and metric history live in MLflow.** The CLI streams
logs and status, not training metrics. The submit/status response includes the
MLflow experiment and run URL; echo it so the user can open the run in the
Databricks UI for loss curves, GPU utilization, and system charts.

**Loss、GPU 利用率和指标历史记录位于 MLflow。** CLI 会流式传输日志和状态,但不会提供训练指标。提交和状态响应包含 MLflow experiment 和 run URL;将其回显出来,以便用户可以在 Databricks UI 中打开该运行,查看 loss 曲线、GPU 利用率和系统图表。

If a run fails

运行失败时

Get the failure reason, then pull the logs and surface the actual error to the user:
bash
databricks air get <JOB_RUN_ID> -o json -p <profile>             # termination_reason
databricks air logs <JOB_RUN_ID> --tail 500 -p <profile>         # latest attempt
databricks air logs <JOB_RUN_ID> --retry 0 --tail 500 -p <profile>   # first attempt
If the run was retried, the latest attempt's logs may show a different symptom than the first failure; check
--retry 0
. On a multi-node run, check other nodes with
--node N
when node 0 looks healthy.
Show the failing traceback and let the user decide the fix. Before resubmitting, set
max_retries: 0
so a deterministic error fails once. For a deeper, guided diagnosis of a training failure, use AI Runtime's own debugging support rather than guessing at root cause here.
获取失败原因,然后拉取日志并向用户呈现实际错误:
bash
databricks air get <JOB_RUN_ID> -o json -p <profile>             # termination_reason
databricks air logs <JOB_RUN_ID> --tail 500 -p <profile>         # latest attempt
databricks air logs <JOB_RUN_ID> --retry 0 --tail 500 -p <profile>   # first attempt
如果运行经过重试,最新尝试的日志可能会显示与首次失败不同的症状;请检查
--retry 0
。对于多节点运行,当节点 0 看起来正常时,请使用
--node N
检查其他节点。
显示失败的 traceback,并让用户决定如何修复。重新提交前,将
max_retries
设为 0,使确定性错误只失败一次。若需对训练失败进行更深入的引导式诊断,请使用 AI Runtime 自带的调试支持,而不是在这里猜测根本原因。

Serving a trained model

为训练后的模型提供服务

A model trained with
databricks air
is served as a custom model: log it with MLflow, register it in Unity Catalog, and create a serving endpoint on CPU or GPU compute.
To serve it with the library versions you tested, without pinning
torch
or CUDA for the serving container, use an express deployment (Public Preview). The endpoint then runs the environment of the notebook you registered from. Express deployments require:
  • Logging and registering from an AI Runtime notebook to serve on GPU, or from a serverless notebook to serve on CPU, with
    mlflow>=3.12
    and
    databricks-sdk>=0.102.0
    .
  • Registering with
    mlflow.register_model(..., env_pack="databricks_model_serving")
    . Do not use
    registered_model_name=
    on
    log_model
    : it does not accept
    env_pack
    and registers a standard model.
  • An environment of at most 200 GB.
For LLMs, use Custom LLM Serving, which builds on express deployments and starts an OpenAI-compatible server such as vLLM from the model's
entrypoint
metadata.
GPU
workload_type
names map to different GPUs per cloud. For example,
GPU_LARGE
is an L40S (48 GB) on AWS and an A100 (80 GB) on Azure. See the GPU table on the Custom LLM Serving page.
使用
databricks air
训练的模型可作为 custom model 提供服务: 将其记录到 MLflow,在 Unity Catalog 中注册,并在 CPU 或 GPU compute 上创建 serving endpoint。
要在不固定 serving 容器中
torch
或 CUDA 版本的情况下,使用已测试的库版本提供服务,请使用 express deployment (Public Preview)。该 endpoint 随后会运行你注册来源 notebook 的环境。Express deployments 要求:
  • 从 AI Runtime notebook 记录并注册以在 GPU 上提供服务,或从 serverless notebook 记录并注册以在 CPU 上提供服务,并使用
    mlflow>=3.12
    和
    databricks-sdk>=0.102.0
    。
  • 使用
    mlflow.register_model(..., env_pack="databricks_model_serving")
    进行注册。不要在
    log_model
    上使用
    registered_model_name=
    :它不接受
    env_pack
    ,并且会注册标准模型。
  • 环境大小不超过 200 GB。
对于 LLM,请使用 Custom LLM Serving, 它基于 express deployments 构建,可根据模型的
entrypoint
元数据启动 OpenAI 兼容服务器,例如 vLLM。
GPU
workload_type
名称会映射到不同云上的不同 GPU。例如,
GPU_LARGE
在 AWS 上对应 L40S(48 GB),在 Azure 上对应 A100(80 GB)。请参阅 Custom LLM Serving 页面上的 GPU 表格。

Custom Docker images

自定义 Docker 镜像

Use a custom Docker image instead of
environment.version
when your workload needs specific system libraries, CUDA extensions (flash-attn, apex, custom kernels), or dependencies that do not fit
environment.dependencies
.
Read docker-images.md for step-by-step guidance on:
  • Using Databricks-provided base images, including CUDA 13 images for B300
  • Dockerfile patterns
  • Pre-build compatibility checklist (CUDA/driver, PyTorch, NCCL, EFA/RDMA)
  • Pushing images with
    databricks air images push
    and running them with
    environment.unity_catalog_image
  • Troubleshooting runtime errors such as
    ssl.SSLError
    on FIPS hosts
当工作负载需要特定的系统库、CUDA 扩展(flash-attn、apex、自定义 kernel),或无法通过
environment.dependencies
安装的依赖时,请使用自定义 Docker 镜像代替
environment.version
。
有关以下内容的分步指南,请阅读 docker-images.md:
  • 使用 Databricks 提供的基础镜像,包括适用于 B300 的 CUDA 13 镜像
  • Dockerfile 模式
  • 预构建兼容性检查清单(CUDA/driver、PyTorch、NCCL、EFA/RDMA)
  • 使用
    databricks air images push
    推送镜像,并通过
    environment.unity_catalog_image
    运行镜像
  • 排查运行时错误,例如 FIPS 主机上的
    ssl.SSLError