Databricks AI Runtime ()
Databricks AI Runtime is a set of
CLI commands for submitting GPU training workloads to Databricks serverless compute. It manages environment setup, distributed training configuration, and workload lifecycle, without requiring you to manage clusters or infrastructure.
This skill covers the three everyday flows: starting a run, checking on a run, and monitoring a run as it trains.
Reference discipline
The CLI's built-in help is the source of truth for field names, accelerator
types, defaults, and constraints. Prefer it at runtime over anything memorized.
Look up a config field by passing its path to
on the
command (the
path is a separate argument):
bash
databricks air run -h config # list the YAML fields
databricks air run -h config.compute # one field
databricks air run -h config.compute.accelerator_type # a nested field
databricks air <subcommand> --help
documents each command's flags. This skill
covers workflow and the few things the help does not spell out.
Session setup
Check the CLI version, confirm the command is available, and list your
Databricks config profiles:
bash
databricks version
databricks air --help
databricks auth profiles --skip-validate # available Databricks config profiles
The examples use flattened
fields, available in
Databricks CLI 1.20.0 and newer.
On an older CLI, use the equivalent nested snapshot format shown under
Workload YAML. CLI 1.20.0 and newer accept both formats.
Do not mix nested and flattened fields in the same configuration.
Pass
on every command. Add
for scripted/programmatic
calls so you get a structured envelope instead of human text:
- Success:
{"v": 1, "ts": "...", "data": {...}}
- Error:
{"v": 1, "ts": "...", "error": {"code": "...", "kind": "...", "message": "...", "retryable": bool}}
Config validation (input) errors are the exception: they print as plain text to
stderr with a non-zero exit even under
, so check the exit code and
stderr too, not just the parsed JSON. When you do get a structured error, its
tells you what to do:
| kind | Meaning | Action |
|---|
| Not authenticated / permission denied / feature disabled / non-retryable | If auth: databricks auth login --host <workspace-url>
|
| Bad run ID / resource | Check the input |
| Idempotency key collision | Use a new |
| Retryable | Retry |
Subcommands
| Command | What it does |
|---|
| Submit a workload from a YAML file. Add to stream logs until it finishes, to validate without submitting, and --override <field>=<value>
(repeatable) to change a field without editing the YAML. |
databricks air get <JOB_RUN_ID>
| Show status, config, timing, and the failure reason for one run. |
| List your active runs. A run that just finished drops out of the default view; add to include finished runs. Also , , . |
databricks air cancel <JOB_RUN_ID> [<JOB_RUN_ID>...]
/ | Cancel one or more runs, or all of your active runs ( prompts; add in scripts). |
databricks air logs <JOB_RUN_ID>
| Stream a running run's logs, or fetch a finished run's logs. |
databricks air pools list
/ | View the provisioned GPU pools available to the workspace. |
databricks air images push
| Push a local or remote container image to Databricks Artifact Registry. See docker-images.md. |
databricks air convert-to-dabs <yaml_path>
| Convert a workload YAML into a Databricks Asset Bundle (local only; deploy with ). |
For flags and defaults, run
databricks air <subcommand> --help
.
Workload YAML
yaml
experiment_name: my-training-job # alphanumeric, - and _ only
compute:
num_accelerators: 1
accelerator_type: GPU_1xA10 # run `databricks air run -h config.compute` for valid values
environment:
version: "6" # Standard environment; use databricks_ai_v6 for PyTorch preinstalled
dependencies:
- torch
- mlflow
code_source:
root_path: "." # relative to workload.yaml; extracted to $CODE_SOURCE_PATH on each node
max_retries: 0 # while iterating; see Retries below
command: |-
cd "$CODE_SOURCE_PATH"
python train.py
Submit with
databricks air run --file workload.yaml -p <profile>
. Field
details live in
databricks air run -h config.<field>
(the authoritative
source).
For an older CLI, replace only the
block above with:
yaml
code_source:
type: snapshot
snapshot:
root_path: "."
The values and behavior are the same: add
under
and put all snapshot fields (
, and any
,
, or
) inside
. Keep the rest of the workload
unchanged. Check
databricks air run -h config.code_source
for the fields
supported by the installed CLI.
Accelerator types
Run
databricks air run -h config.compute
to see the accelerator types. On CLI
1.23.0 and newer it lists the types
available in the current workspace and
how many GPUs each type has per node; if it cannot reach the workspace, it
prints a warning and falls back to a built-in list. Do not guess a type from
memory: availability depends on the workspace, cloud, and region, and matching
is case-sensitive.
Type names follow
GPU_<GPUs per node>x<GPU>
, for example
,
,
, and
.
is the total
GPU count across all nodes, so it must be a multiple of the per-node count: 8
on an
type is one node, 16 is two nodes. Only the
types support
multi-GPU and multi-node training.
Python environment
Pick the environment from what the training code imports, before you submit,
so a missing package does not surface only after a GPU has been provisioned:
| What it provides | Use when |
|---|
| (the default) | Standard environment: Python and CUDA only. Since version 5, it does not include or . | You control the dependency stack. List every import, including , under . |
version: databricks_ai_v6
| Databricks AI environment: PyTorch, Transformers, and other ML frameworks preinstalled. | Typical PyTorch or Hugging Face training. Add extras under . |
unity_catalog_image: <catalog>.<schema>.<image>:<tag>
| Your own Docker image. | System libraries or CUDA extensions; see Custom Docker images. |
Before submitting, check the script's imports against the chosen environment.
For the exact packages in each environment version, see the
environment version release notes.
Do not combine
with
or
.
Configuring the workload
These optional top-level fields cover the common needs. Run
databricks air run -h config.<field>
for the full reference on any of them.
Hugging Face gated models
When a model needs a Hugging Face (HF) token, lead with this checklist before
asking for other training details or running secrets commands:
- Accept the model license on HF and, if required, wait for access approval.
Create a fine-grained read-only token with Read access to contents of all
public gated repos you can access. For a private repo, also grant read
access to that specific repo.
- Choose an existing scope the user owns and can write. If they need to find
one, have them filter to their own username or scopes they
created, then confirm ownership and access; a name match alone is not proof.
For example, the user can run the following with their own username:
bash
databricks secrets list-scopes -p <profile> -o json | jq -r --arg user "<your-username>" '.[] | select(.name | contains($user)) | .name'
Never browse or inspect other users' scopes or keys. Create a scope only if
needed; if reports Max limit of 2000 scopes has been reached
,
reuse an existing user-owned scope instead. The user may need to run
on their own scope to check access. Before using in any
command, validate it: it must be an existing scope
the user owns, from their own scopes, and match the scope-name charset
. If it doesn't, have the user run the commands
themselves. Always double-quote it. If the scope is Azure Key Vault-backed
( is ), can't write to it: have
the user add the token to that vault with their own tooling and use that
secret's name as the key below.
- First check whether already exists in the scope, using the
key-name check in step 4. If it does, reuse it only if it is the right
token; otherwise pick a new key such as , or get the user's
explicit OK before overwriting it, since other workloads may read it. Use
the chosen key in steps 4 and 5. Have the user run this in their own
terminal and enter the token at the hidden prompt. Never paste the token
into chat or a command line.
bash
databricks secrets put-secret "<scope>" hf-token -p <profile>
- Once they have stored it and named their scope, verify only the key name in
that one scope, filtering the output to :
bash
databricks secrets list-secrets "<scope>" -p <profile> -o json | jq -r '.[] | select(.key == "hf-token") | .key'
- Reference
secrets: {HF_TOKEN: "<scope>/hf-token"}
in the workload. Quote
the reference, particularly when the scope contains or . Transformers
reads automatically.
Gemma 3 is gated; Mistral Small 3.2, gpt-oss-20b, and Qwen3 are ungated and do
not need a token just to download their public weights. Check the specific
model's HF page for current access requirements.
Environment variables and secrets
Set plain values under
; pull sensitive values from Databricks
Secrets under
(each entry maps an env var name to a
, and
the value is materialized on the worker, never logged):
yaml
env_variables:
BATCH_SIZE: "32"
secrets:
HF_TOKEN: "<scope>/hf-token" # -> $HF_TOKEN on every node
WANDB_API_KEY: "<scope>/wandb_key"
The scope and keys must already exist. Prefer a user-owned scope; create one
only if needed, and have the user store secret values in their own terminal.
Secret env var names must not collide with
names.
Parameters (hyperparameters)
is the clean way to pass hyperparameters into your script and
change them between runs without editing training code. The CLI writes them to
a YAML file on the cluster and exposes its path as
:
yaml
parameters:
learning_rate: 0.001
batch_size: 32
warmup_steps: 1000
use_amp: true
Read them in the script:
python
import os, yaml
with open(os.environ["HYPERPARAMETERS_PATH"]) as f:
hp = yaml.safe_load(f)
lr = hp["learning_rate"]
Values can be strings, numbers, booleans, lists, or nested dicts. To sweep a
hyperparameter without editing the YAML, resubmit with
--override parameters.learning_rate=0.0005
(and a fresh idempotency key).
Retries
Failed attempts are retried automatically:
defaults to 3 when
unset (confirm with
databricks air run -h config.max_retries
). Each retry
provisions GPUs and reruns the whole command, so a deterministic failure (an
import error, a typo, a bad path) fails 4 times before the run gives up.
Set while writing, debugging, or iterating on a workload,
either in the YAML or with
. Raise it only for a
long, stable training run where retrying can recover from transient
infrastructure failures, and make the training code resume from a checkpoint
so a retry does not start over. When
is set, it applies
per attempt.
Permissions
By default only you can see the run. Grant others access to the submitted job
with
(DABs-compatible format); levels are
,
,
:
yaml
permissions:
- group_name: data-engineering
level: CAN_VIEW
- user_name: alice@example.com
level: CAN_MANAGE
Other fields
| Field | Use |
|---|
| Wall-clock limit per attempt, in minutes. |
| YAML equivalent of : resubmitting with the same token returns the existing run. |
, mlflow_experiment_directory
| Name the MLflow run, or choose the workspace directory that holds the experiment. |
| , | Run on a pre-provisioned GPU pool (find IDs with databricks air pools list
). is or (preemptable) and requires . |
| Run different images and commands on different node ranks, for example a trainer and an inference server. Replaces ; each entry needs , , , and . See databricks air run -h config.containers
. |
Environment variables provided on each node
sets these automatically on every node; use them in
and in your training code.
| Variable | Meaning |
|---|
| Absolute path to your extracted on the node (). |
| / | Address/port of the rank-0 node for distributed coordination. |
| Total number of nodes. |
| Rank of the current node (0 to -1). Also set as . |
| Total number of GPUs across all nodes. |
| Number of GPUs on this node. |
| MLflow run ID for this job (log metrics/artifacts to it). |
| Path to the YAML file (only set when is defined). |
Your
runs
once per node, so per-process ranks (
,
) are
not set by the platform. Your distributed launcher
(torchrun, accelerate) assigns them when it spawns one process per GPU; pass it
/
/
/
.
Starting a run
Start from a cookbook recipe when one fits. The
AI Runtime Cookbook
has runnable recipes, each a
plus source code, grouped by domain
(
,
,
recommendation-and-ranking/
,
,
,
, and more; Ray recipes live in these
domain folders with a
prefix).
is the
minimal first run. Adapting the closest recipe is usually faster and less
error-prone than writing a workload from scratch:
bash
git clone https://github.com/databricks-solutions/ai-runtime-cookbook.git
cd ai-runtime-cookbook/<domain>/<recipe> # e.g. getting-started/air-cli
# Review workload.yaml (and the recipe README) for values to configure, then:
databricks air run --file workload.yaml --dry-run -o json -p <profile>
databricks air run --file workload.yaml --watch \
--idempotency-key <key> -o json -p <profile> # key as in step 5
Apply the steps below to the recipe's YAML the same way as to a new one,
including the stable
from step 5.
-
Turn the user's intent into the YAML. Extract model, dataset path, GPU
count and type, and the training command from the request. For gated models,
lead with the
token checklist first. Ask before
synthesizing a config when a remaining critical detail is ambiguous (data
path, checkpoint dir, custom image).
-
Write the using the env vars above. Common launch shapes:
| Framework | shape |
|---|
| plain torch, single GPU | |
| torch, multi-GPU (1 node) | torchrun --nproc_per_node=$LOCAL_WORLD_SIZE train.py
|
| torch, multi-node | torchrun --nnodes=$NUM_NODES --nproc_per_node=$LOCAL_WORLD_SIZE --node_rank=$NODE_RANK --master_addr=$MASTER_ADDR --master_port=$MASTER_PORT train.py
|
| HuggingFace Accelerate | accelerate launch --config_file accelerate.yaml train.py
|
| DeepSpeed | deepspeed --num_gpus=$LOCAL_WORLD_SIZE train.py
|
| Composer / MosaicML | |
Multi-line commands go under
. Access uploaded code via
.
Total GPUs = nodes x accelerators-per-node; see
Accelerator types.
-
Pre-flight the referenced resources so the job does not die on startup:
- Every package the script imports is in the chosen
environment or its (the Standard
environment has no ).
- is set while the workload is new or being debugged.
- For each reference, verify only its key name in the user's chosen
scope, filtering as in the token checklist.
Never browse other users' scopes or list their keys.
- If
environment.unity_catalog_image
is set, the image has been pushed to
Databricks Artifact Registry in the same workspace and the submitting
identity can read it (see docker-images.md).
-
Dry-run before submitting. This validates the config without launching:
bash
databricks air run --file workload.yaml --dry-run -o json -p <profile>
On CLI 1.20.0 and newer, the dry run also checks the config against the
workspace, so it needs a working profile. A config-validation error is a
hard stop: it prints as plain text on stderr with a non-zero exit, so show
that message to the user and fix the YAML. Other failures are infra/auth
and should bubble up.
-
Submit. Use a stable
so an accidental re-run does not
double-submit (compose it from
<user>-<experiment>-<YYYYMMDD>-v<N>
and bump
on each resubmit; avoid dynamic subexpressions like
so the
command stays auto-approvable):
bash
databricks air run --file workload.yaml \
--idempotency-key <key> -o json -p <profile>
Add
to stream logs to your terminal until the run finishes. Capture
from the response, then monitor it.
Checking on a run
bash
databricks air get <JOB_RUN_ID> -o json -p <profile> # status, config, timing, failure reason
databricks air list -o json -p <profile> # your active runs
databricks air list --all-status -o json -p <profile> # include finished runs
reports
while the run waits for GPUs, then
, then
a final state such as
or
. An empty
means you have
no
active runs, not that the run is gone; use
or
to see a finished run.
Monitoring a run
Stream logs while it runs. The simplest path is to submit with
,
which streams the run's logs to your terminal until it reaches a terminal state:
bash
databricks air run --file workload.yaml --watch -p <profile>
For a run that is already going, stream its logs by ID:
bash
databricks air logs <JOB_RUN_ID> -p <profile>
# --node N (one node), --tail N (last N lines of a finished run),
# --minutes N (last N minutes), --retry N (one retry attempt, 0-indexed),
# --download-to DIR (save all nodes' logs)
Loss, GPU utilization, and metric history live in MLflow. The CLI streams
logs and status, not training metrics. The submit/status response includes the
MLflow experiment and run URL; echo it so the user can open the run in the
Databricks UI for loss curves, GPU utilization, and system charts.
If a run fails
Get the failure reason, then pull the logs and surface the actual error to the
user:
bash
databricks air get <JOB_RUN_ID> -o json -p <profile> # termination_reason
databricks air logs <JOB_RUN_ID> --tail 500 -p <profile> # latest attempt
databricks air logs <JOB_RUN_ID> --retry 0 --tail 500 -p <profile> # first attempt
If the run was retried, the latest attempt's logs may show a different symptom
than the first failure; check
. On a multi-node run, check other
nodes with
when node 0 looks healthy.
Show the failing traceback and let the user decide the fix. Before
resubmitting, set
so a deterministic error fails once. For a
deeper, guided diagnosis of a training failure, use AI Runtime's own debugging
support rather than guessing at root cause here.
Serving a trained model
A model trained with
is served as a
custom model:
log it with MLflow, register it in Unity Catalog, and create a serving endpoint
on CPU or GPU compute.
To serve it with the library versions you tested, without pinning
or
CUDA for the serving container, use an
express deployment
(Public Preview). The endpoint then runs the environment of the notebook you
registered from. Express deployments require:
- Logging and registering from an AI Runtime notebook to serve on GPU, or from
a serverless notebook to serve on CPU, with and
.
- Registering with
mlflow.register_model(..., env_pack="databricks_model_serving")
. Do not use
on : it does not accept and
registers a standard model.
- An environment of at most 200 GB.
For LLMs, use
Custom LLM Serving,
which builds on express deployments and starts an OpenAI-compatible server such
as vLLM from the model's
metadata.
GPU
names map to different GPUs per cloud. For example,
is an L40S (48 GB) on AWS and an A100 (80 GB) on Azure. See the GPU
table on the Custom LLM Serving page.
Custom Docker images
Use a custom Docker image instead of
when your workload
needs specific system libraries, CUDA extensions (flash-attn, apex, custom
kernels), or dependencies that do not fit
.
Read docker-images.md for step-by-step guidance on:
- Using Databricks-provided base images, including CUDA 13 images for B300
- Dockerfile patterns
- Pre-build compatibility checklist (CUDA/driver, PyTorch, NCCL, EFA/RDMA)
- Pushing images with
databricks air images push
and running them with
environment.unity_catalog_image
- Troubleshooting runtime errors such as on FIPS hosts