serve-model

Compare original and translation side by side

🇺🇸

Original

English

🇨🇳

Translation

Chinese

Serve a model with MAX

使用MAX部署模型

max serve

launches an OpenAI-compatible HTTP server for a model. It handles tokenization, batching, KV cache, and the whole serving stack. You point it at a checkpoint and, if the model isn't built into MAX, at a custom architecture package. This skill takes you from "no environment" to "server answering requests," and helps you choose flags that fit the specific model instead of guessing.

The guiding principle: start from the smallest command that could work, then add flags only when the model or the hardware forces you to. MAX auto-detects most things (dtype, sequence length, device defaults). Over-specifying flags is the most common way people turn a working serve into a broken one.

Use this skill when you want to run, launch, or host a model on MAX: bring up an OpenAI-compatible endpoint, serve a built-in or a custom/ported architecture, or debug a

max serve

startup failure.

Do not use this skill when the model isn't implemented in MAX yet (no working

arch.py

, graph, and weights). That's a bring-up task: use

import-model

to port the architecture, and

debug-model

if it serves but the output is wrong. This skill runs an existing model; it doesn't author one.

max serve

会为模型启动一个兼容OpenAI的HTTP服务器，负责处理分词、批处理、KV缓存及整个部署栈。只需将其指向一个checkpoint，若模型未内置到MAX中，再指向自定义架构包即可。本指南将带你从“无环境”状态到“服务器响应请求”，并帮助你为特定模型选择合适的参数，而非盲目猜测。

核心原则：从最简化的可行命令开始，仅当模型或硬件要求时才添加参数。MAX会自动检测大多数配置（数据类型、序列长度、设备默认值）。过度指定参数是导致原本可正常运行的部署失败的最常见原因。

适用场景：当你想要在MAX上运行、启动或托管模型时：搭建OpenAI兼容端点、部署内置或自定义/移植架构，或调试

max serve

启动失败问题。

不适用场景：模型尚未在MAX中实现（无可用的

arch.py

、计算图和权重）。这种情况属于模型适配任务：使用

import-model

移植架构，若已部署但输出错误则使用

debug-model

。本指南仅用于运行现有模型，不涉及模型开发。

References

参考文档

File	Read when
references/custom-arch.md	Serving a custom architecture: the `arch.py` -to-flags mapping, encoding and device, and serve-time gotchas
references/flags.md	Choosing any serve flag beyond `--model` , `--devices` , `--quantization-encoding` , and `--max-length`
references/troubleshooting.md	A `max serve` startup failure or a cryptic error

Read the reference for what you're doing, not all of them upfront.

文件路径	阅读时机
references/custom-arch.md	部署自定义架构时： `arch.py` 与参数的映射关系、编码与设备配置，以及部署时的常见陷阱
references/flags.md	选择 `--model` 、 `--devices` 、 `--quantization-encoding` 和 `--max-length` 之外的其他部署参数时
references/troubleshooting.md	遇到 `max serve` 启动失败或晦涩错误时

按需阅读对应参考文档，无需提前通读全部内容。

Fast path (custom architecture): do this first

快速流程（自定义架构）：优先执行以下步骤

If MAX is already installed and you have a working custom-arch package, this is the whole job in four calls. Don't hand-read

arch.py

and

config.json

and reason about flags yourself. The bundled inspector does exactly that and prints a ready-to-run command plus the reasoning:

bash

undefined

若已安装MAX且拥有可用的自定义架构包，只需四步即可完成部署。无需手动阅读

arch.py

和

config.json

并推断参数，内置的检查工具会自动完成这些工作，并输出可直接运行的命令及推理过程：

bash

undefined

1. Get the recommended command + notes (reads arch.py + config.json for you).

1. 获取推荐命令及说明（自动读取arch.py和config.json）。

python <skill>/scripts/suggest_serve_command.py
--custom-architectures /abs/path/to/my_arch --model <hf-repo-or-path>

2. Launch it (add

pixi run

if in a pixi project). On a REMOTE box, wrap with

2. 启动服务器（若在pixi项目中，需添加

pixi run

前缀）。在远程服务器上，使用

setsid ... </dev/null

so it survives the SSH session:

setsid ... </dev/null

确保SSH会话关闭后服务器仍能运行：

setsid <the suggested command> </dev/null > /tmp/max-serve.log 2>&1 &

3. Wait for readiness in ONE call (fails fast on a crash; ~10 min budget for

3. 一键等待服务器就绪（启动失败时快速报错；冷编译最多等待约10分钟，出现

Still compiling

心跳信息为正常现象）：

a cold compile, advancing

Still compiling

heartbeats are normal):

—

timeout 600 bash -c 'until grep -qE "Server ready|Uvicorn running" /tmp/max-serve.log; do grep -qiE "Traceback|CRASHED|Error building|cannot be found|not found in registry" /tmp/max-serve.log && { echo SERVE_FAILED; tail -30 /tmp/max-serve.log; exit 1; } sleep 3; done' && echo SERVE_READY

4. Confirm with one request (model field must equal --served-model-name):

4. 发送请求验证（model字段必须与--served-model-name一致）：

curl -s http://localhost:8000/v1/chat/completions -H "Content-Type: application/json"
-d '{"model":"my_arch","messages":[{"role":"user","content":"The capital of France is"}],"max_completion_tokens":32}'


**Read the inspector's `# notes:`**. That's where the traps surface (a
`default_encoding` that disagrees with the checkpoint, a `name` colliding with a
built-in, GPU-only encodings, MoE). If the command works and the output is
coherent, you're done. Only drop into the detailed steps below when a note or a
failure tells you to. The rest of this doc is the "why" behind what the
inspector does and what to do when it isn't enough.


**务必阅读检查工具输出的`# notes:`部分**，这里会提示潜在陷阱（如`default_encoding`与checkpoint不匹配、名称与内置架构冲突、仅支持GPU的编码、MoE模型等）。若命令可正常运行且输出内容连贯，则部署完成。仅当提示信息或部署失败时，才需要查看下方的详细步骤。本文档剩余部分将解释检查工具的工作原理，以及工具无法解决问题时的处理方法。

1. Make sure MAX is installed

1. 确保已安装MAX

The user needs a

max

binary from the nightly build. Check first, and don't reinstall if it's already there (the

max serve

command works only if the project includes the

max[serve]

max-serve

extra dependencies):

bash

max serve --help           # already in a MAX env?
pixi run max serve --help   # or inside a pixi project

If the

max serve

command isn't available, set up an environment. pixi is the default; the key detail is the conda channel

https://conda.modular.com/max-nightly/

plus

conda-forge

bash

undefined

用户需要来自** nightly 构建**的

max

二进制文件。先检查是否已安装，若已安装则无需重新安装（

max serve

命令仅在项目包含

max[serve]

或

max-serve

额外依赖时可用）：

bash

max serve --help           # 是否已在MAX环境中？
pixi run max serve --help   # 或在pixi项目中执行

若

max serve

命令不可用，则需搭建环境。pixi是默认选择，关键配置是conda源

https://conda.modular.com/max-nightly/

加上

conda-forge

：

bash

undefined

pixi (conda channels)

pixi（conda源）

curl -fsSL https://pixi.sh/install.sh | sh pixi init my-max-project
-c https://conda.modular.com/max-nightly/ -c conda-forge && cd my-max-project pixi add max-serve


If a `pixi.toml` already exists, the channels line must read exactly:

```toml
[workspace]                                # or [project] on older pixi
channels = ["https://conda.modular.com/max-nightly/", "conda-forge"]

Some users prefer uv, which pulls MAX from Modular's wheel index instead of conda:

bash

undefined

curl -fsSL https://pixi.sh/install.sh | sh pixi init my-max-project
-c https://conda.modular.com/max-nightly/ -c conda-forge && cd my-max-project pixi add max-serve


若已存在`pixi.toml`文件，其源配置必须如下：

```toml
[workspace]                                # 旧版pixi为[project]
channels = ["https://conda.modular.com/max-nightly/", "conda-forge"]

部分用户偏好uv，它会从Modular的wheel索引而非conda获取MAX：

bash

undefined

uv (pip wheels)

uv（pip wheels）

curl -LsSf https://astral.sh/uv/install.sh | sh uv init my-max-project && cd my-max-project uv venv && source .venv/bin/activate uv add "max[serve]" --index https://whl.modular.com/nightly/simple/ --prerelease allow


After a pixi setup, prefix commands with `pixi run` (or enter `pixi shell`).
After uv, activate the venv (`source .venv/bin/activate`) and call `max`
directly. The rest of this skill writes bare `max serve ...`; add `pixi run` in
front when you're in a pixi project and haven't entered the shell.


使用pixi搭建环境后，命令需添加`pixi run`前缀（或进入`pixi shell`）。使用uv搭建环境后，需激活虚拟环境（`source .venv/bin/activate`），然后直接调用`max`。本指南后续内容使用`max serve ...`形式，若在pixi项目中且未进入shell，需在命令前添加`pixi run`。

2. Is this a built-in model or a custom architecture?

2. 判断是内置模型还是自定义架构？

MAX ships with many architectures. If the model's architecture is already supported, you don't need

--custom-architectures

at all, just

--model

bash

max list                 # every registered architecture + example repo IDs

Match the checkpoint's

config.json::architectures[0]

(for example

LlamaForCausalLM

Qwen2ForCausalLM

) against that list.

Listed: built-in. Skip to step 4 and omit
```
--custom-architectures
```
.
Not listed: you need a custom architecture package (step 3). If the user doesn't have one yet, this skill can't manufacture it; that's a bring-up task (implementing
```
arch.py
```
,
```
model.py
```
, the graph, weight adapters). Point them at the model bring-up workflow and stop here.

MAX内置了多种架构。若模型架构已被支持，则无需使用

--custom-architectures

，仅需

--model

参数。

bash

max list                 # 查看所有已注册的架构及示例仓库ID

将checkpoint的

config.json::architectures[0]

（例如

LlamaForCausalLM

、

Qwen2ForCausalLM

）与列表进行匹配。

已列出：属于内置架构。跳过步骤3，直接进入步骤4，无需添加
```
--custom-architectures
```
。
未列出：需要自定义架构包（步骤3）。若用户尚未拥有该包，本指南无法创建它，这属于模型适配任务（实现
```
arch.py
```
、
```
model.py
```
、计算图、权重适配器）。请引导用户参考模型适配流程，并停止本操作。

3. Target a custom architecture

3. 指定自定义架构

A custom architecture is a Python package (a directory with

__init__.py

) that exposes a top-level

ARCHITECTURES

list of

SupportedArchitecture

instances. You pass the package with

--custom-architectures

; MAX imports it, registers each arch by

name

, and on each request matches the checkpoint's

config.json::architectures[0]

against a registered

name

Read the package before you write the command. The architecture package is the source of truth for most of the flags, so don't guess them. Open two files:

arch.py
: the
```
SupportedArchitecture(...)
```
call. Its
```
default_encoding
```
is the encoding to serve with;
```
supported_encodings
```
is the set your
```
--quantization-encoding
```
must belong to;
```
name
```
must match the checkpoint;
```
multi_gpu_supported
```
says whether you can shard; a
```
chat_template.jinja
```
in the package means you'll need
```
--chat-template
```
.

config.json
(the checkpoint):

architectures[0]

must equal

arch.py

name

;

max_position_embeddings

caps

--max-length

;

torch_dtype

quantization_config

should agree with

default_encoding

; MoE fields (

num_experts

etc.) hint that you want

--device-graph-capture

references/custom-arch.md
is the detailed guide: a field-by-field

arch.py

-to-flags mapping, the encoding-to-device table, and the non-obvious serve-time gotchas (overlap-scheduler vs logprobs, the port-8001 metrics collision, trust-remote-code). Read it whenever you're serving a custom arch.

Point

--custom-architectures

at the package directory (an absolute path is safest for scripts and remote hosts). The

IMPORT_PATH:MODULE_NAME

colon form also works, but the directory path is what most tooling uses:

bash

max serve --model <hf-repo-or-path> --custom-architectures /abs/path/to/my_arch

Run these three checks up front rather than reading a stack trace; they head off almost every custom-arch serve failure:

name=

in the

SupportedArchitecture

exactly equals

config.json::architectures[0]

```
supported_encodings
```
includes the encoding the checkpoint actually ships (a bf16 checkpoint needs
```
bfloat16
```
, a GPTQ checkpoint needs
```
gptq
```
, etc.).

weight_adapters

has an entry for the checkpoint's weight format (

WeightsFormat.safetensors

for

.safetensors

WeightsFormat.gguf

for

.gguf

If the model isn't already a working custom-arch package (the graph, weight adapters, and config aren't implemented yet), that's a bring-up task, not a serving one. This skill serves an existing package; it doesn't author one.

自定义架构是一个Python包（包含

__init__.py

的目录），暴露顶层

ARCHITECTURES

列表，其中包含

SupportedArchitecture

实例。通过

--custom-architectures

参数传入该包，MAX会导入它，按

name

注册每个架构，并在每次请求时将checkpoint的

config.json::architectures[0]

与已注册的

name

进行匹配。

编写命令前先阅读包内容。架构包是大多数参数的权威来源，请勿盲目猜测。打开以下两个文件：

arch.py
：
```
SupportedArchitecture(...)
```
调用。其中
```
default_encoding
```
为部署时使用的编码；
```
supported_encodings
```
是
```
--quantization-encoding
```
必须属于的集合；
```
name
```
必须与checkpoint匹配；
```
multi_gpu_supported
```
表示是否支持分片；若包中包含
```
chat_template.jinja
```
，则需要添加
```
--chat-template
```
参数。

config.json
（checkpoint文件）：

architectures[0]

必须等于

arch.py

中的

name

；

max_position_embeddings

限制

--max-length

的最大值；

torch_dtype

quantization_config

应与

default_encoding

一致；MoE相关字段（

num_experts

等）提示需要使用

--device-graph-capture

。

references/custom-arch.md
是详细指南：包含

arch.py

字段与参数的映射表、编码与设备对应表，以及部署时的非明显陷阱（重叠调度器vs对数概率、8001端口指标冲突、信任远程代码）。部署自定义架构时请务必阅读。

将

--custom-architectures

指向包的目录（脚本和远程主机使用绝对路径最安全）。也支持

IMPORT_PATH:MODULE_NAME

格式，但目录路径是大多数工具的常用方式：

bash

max serve --model <hf-repo-or-path> --custom-architectures /abs/path/to/my_arch

提前执行以下三项检查，可避免几乎所有自定义架构部署失败，无需等到查看堆栈跟踪：

SupportedArchitecture

中的

name=

必须完全等于

config.json::architectures[0]

。

```
supported_encodings
```
包含checkpoint实际使用的编码（bf16格式的checkpoint需要
```
bfloat16
```
，GPTQ格式的checkpoint需要
```
gptq
```
等）。

weight_adapters

包含checkpoint权重格式的对应项（

.safetensors

对应

WeightsFormat.safetensors

，

.gguf

对应

WeightsFormat.gguf

）。

若模型尚未成为可用的自定义架构包（未实现计算图、权重适配器和配置），则属于模型适配任务，而非部署任务。本指南仅用于部署现有包，不涉及包的开发。

4. Build the serve command (default-first)

4. 构建部署命令（优先使用默认配置）

Start with the minimal command and run it. Let MAX auto-detect the rest.

bash

max serve --model <hf-repo-or-path> [--custom-architectures ...]

That already binds

0.0.0.0:8000

, serves on the GPU when one is present (CPU otherwise), infers dtype from the checkpoint, and sets

max_length

from

max_position_embeddings

. GPU is the default, so you don't pass

--devices

for the common single-GPU case. For a lot of models on a single GPU, that minimal command is the whole job.

Add flags only for a concrete reason. The three you'll reach for most:

Flag	Add it when	Example
`--devices`	You must pin specific GPUs, shard across GPUs, or force CPU (GPU is already the default).	`--devices gpu:0` · `--devices gpu:0,1,2,3` · `--devices gpu:all` · `--devices cpu`
`--quantization-encoding`	The repo has multiple formats, or auto-detect picks the wrong one.	`--quantization-encoding bfloat16`
`--max-length`	You want a shorter context than the model's max (saves KV memory) or need to cap it to fit.	`--max-length 4096`

For everything else (device memory, batch size, task selection, sliding window, trust-remote-code, multi-GPU parallelism, speculative decoding) see references/flags.md
. Read it before adding any flag you're unsure about; it explains what each one does and when not to set it.

How to decide, in order:

Default first. Try the minimal command. Auto-detection is usually right for built-ins, and GPU is the default device.
For a custom arch, read the package.
```
arch.py
```
's
```
default_encoding
```
is your
```
--quantization-encoding
```
. That encoding constrains the device: fp8, fp4, and gptq are GPU-only, and GPU is already the default, so you don't add
```
--devices
```
for them (see
```
references/custom-arch.md
```
). Cap
```
--max-length
```
at
```
config.json::max_position_embeddings
```
. Add
```
--trust-remote-code
```
if the checkpoint ships custom modeling files, and
```
--chat-template
```
if the package bundles one. These aren't guesses; you read them off the package and config.
Inspect further when a choice is load-bearing. For unusual properties (sliding window, partial RoPE, MoE routing), check
```
config.json
```
and map findings to flags with
```
references/flags.md
```
.
Ask when unsure. If a decision depends on something you can't see (which GPUs are free, how much context they need, CPU vs GPU), ask the user rather than guessing. A wrong
```
--devices
```
or dtype fails slowly and confusingly; a quick question is cheaper.

从最简命令开始运行，让MAX自动检测其余配置。

bash

max serve --model <hf-repo-or-path> [--custom-architectures ...]

该命令已绑定

0.0.0.0:8000

，当存在GPU时默认使用GPU（否则使用CPU），从checkpoint推断数据类型，并根据

max_position_embeddings

设置

max_length

。GPU是默认设备，因此在常见的单GPU场景下无需传入

--devices

。对于许多单GPU上的模型，最简命令即可完成部署。

仅在有明确理由时才添加参数。最常用的三个参数如下：

参数	添加时机	示例
`--devices`	必须固定特定GPU、跨GPU分片，或强制使用CPU（GPU已为默认设备）。	`--devices gpu:0` · `--devices gpu:0,1,2,3` · `--devices gpu:all` · `--devices cpu`
`--quantization-encoding`	仓库包含多种格式，或自动检测选择了错误的编码。	`--quantization-encoding bfloat16`
`--max-length`	需要比模型最大上下文更短的上下文（节省KV内存），或需要限制上下文长度以适配硬件。	`--max-length 4096`

其他参数（设备内存、批处理大小、任务选择、滑动窗口、信任远程代码、多GPU并行、 speculative decoding）请查看**

references/flags.md

**。添加不确定的参数前请先阅读该文档，它会解释每个参数的作用及不应设置的场景。

决策顺序：

优先使用默认配置。尝试最简命令。对于内置模型，自动检测通常是正确的，且GPU为默认设备。
自定义架构需阅读包内容。
```
arch.py
```
中的
```
default_encoding
```
即为
```
--quantization-encoding
```
的取值。该编码会限制设备选择：fp8、fp4和gptq仅支持GPU，而GPU已为默认设备，因此无需添加
```
--devices
```
（详见
```
references/custom-arch.md
```
）。将
```
--max-length
```
限制为
```
config.json::max_position_embeddings
```
的值。若checkpoint包含自定义建模文件，则添加
```
--trust-remote-code
```
；若包中包含聊天模板，则添加
```
--chat-template
```
。这些参数无需猜测，直接从包和配置中读取即可。
关键选择时进一步检查。对于特殊属性（滑动窗口、部分RoPE、MoE路由），查看
```
config.json
```
并参考
```
references/flags.md
```
映射为对应的参数。
不确定时询问用户。若决策依赖于不可见的信息（如可用GPU、所需上下文长度、CPU vs GPU选择），请询问用户而非猜测。错误的
```
--devices
```
或数据类型会导致缓慢且令人困惑的失败，快速询问用户更为高效。

5. Launch and confirm it works

5. 启动并验证服务器正常运行

Launch (backgrounded, with a log you can tail):

bash

max serve --model <hf-repo-or-path> [flags] > /tmp/max-serve.log 2>&1 &

If you're launching on a remote box over SSH, a bare

dies when the SSH session closes, and the compile can outlast your connection. Fully detach it:

bash

setsid max serve --model <hf-repo-or-path> [flags] </dev/null > /tmp/max-serve.log 2>&1 &

Wait for readiness with a single call that watches the log's heartbeat and fails fast on a crash instead of blocking for the full timeout:

bash

timeout 600 bash -c 'until grep -qE "Server ready|Uvicorn running" /tmp/max-serve.log; do
  grep -qiE "Traceback|CRASHED|Error building|cannot be found|not found in registry" /tmp/max-serve.log && { echo SERVE_FAILED; tail -30 /tmp/max-serve.log; exit 1; }
  sleep 3; done' && echo SERVE_READY

The server prints this line when it's ready:

output

Server ready on http://0.0.0.0:8000 (Press CTRL+C to quit)

Large models compile on first launch. A quiet gap with

Still compiling model (Ns elapsed)

heartbeats is normal, not a hang: the elapsed counter is the liveness signal. Wait while it advances, and only treat the serve as stuck if the counter freezes (or the log's mtime stops moving while the process is alive).

Confirm health, then send a real request:

bash

curl -s http://localhost:8000/v1/health          # 200 when ready
curl -s http://localhost:8000/v1/models           # served model name

curl -X POST http://localhost:8000/v1/chat/completions \
  -H "Content-Type: application/json" \
  -d '{
    "model": "<hf-repo-or-path>",
    "messages": [{"role": "user", "content": "The capital of France is"}],
    "max_completion_tokens": 32
  }'

The

model

field in the request must match what you passed to

--model

(or

--served-model-name

if you overrode it). Read the response: a bring-up isn't done just because the server is green. Check that the text is actually coherent, not repetition or gibberish. If it serves but the output is wrong, that's a parity/correctness problem, not a serving problem.

启动服务器（后台运行，日志输出到指定文件）：

bash

max serve --model <hf-repo-or-path> [flags] > /tmp/max-serve.log 2>&1 &

若通过SSH在远程服务器上启动，仅使用

会导致SSH会话关闭后服务器终止，且编译过程可能超过会话时长。需完全分离进程：

bash

setsid max serve --model <hf-repo-or-path> [flags] </dev/null > /tmp/max-serve.log 2>&1 &

使用以下命令一键等待服务器就绪，该命令会监控日志心跳，启动失败时快速报错，而非一直阻塞直到超时：

bash

timeout 600 bash -c 'until grep -qE "Server ready|Uvicorn running" /tmp/max-serve.log; do
  grep -qiE "Traceback|CRASHED|Error building|cannot be found|not found in registry" /tmp/max-serve.log && { echo SERVE_FAILED; tail -30 /tmp/max-serve.log; exit 1; }
  sleep 3; done' && echo SERVE_READY

服务器就绪时会输出以下内容：

output

Server ready on http://0.0.0.0:8000 (Press CTRL+C to quit)

大型模型首次启动时会进行编译。出现

Still compiling model (Ns elapsed)

心跳信息的静默间隔是正常现象，并非挂起：计时计数器是存活信号。等待计数器推进，仅当计数器停止（或进程存活但日志修改时间停止）时才认为部署挂起。

验证服务器健康状态，然后发送实际请求：

bash

curl -s http://localhost:8000/v1/health          # 就绪时返回200
curl -s http://localhost:8000/v1/models           # 查看已部署模型名称

curl -X POST http://localhost:8000/v1/chat/completions \
  -H "Content-Type: application/json" \
  -d '{
    "model": "<hf-repo-or-path>",
    "messages": [{"role": "user", "content": "The capital of France is"}],
    "max_completion_tokens": 32
  }'

请求中的

model

字段必须与

--model

参数的值一致（若使用

--served-model-name

覆盖，则需与该值一致）。查看响应内容：仅服务器状态正常并不代表部署完成，需确认输出内容连贯，而非重复或乱码。若服务器可运行但输出错误，属于一致性/正确性问题，而非部署问题。

Cache weights and compilation for faster re-serves

缓存权重和编译结果以加快后续部署速度

The first serve of a model downloads its weights and compiles the graph, which is the slow part. Both results are cached, so later serves of the same model are much faster.

Weights download through
```
huggingface_hub
```
into the shared Hugging Face cache (
```
~/.cache/huggingface
```
by default; set
```
HF_HOME
```
to relocate it). Re-serving the same repo reuses the cached weights with no re-download.
Compilation is cached too. To warm both caches ahead of time (before a demo or deployment) so the first
```
max serve
```
skips the download and the
```
Still compiling
```
wait, run
```
max warm-cache
```
first:
bash
```
max warm-cache --model <hf-repo-or-path> [--custom-architectures /abs/path/to/my_arch]
```
The compiled artifact (MEF) is platform-specific, so warm the cache on the same hardware (or pass
```
--target
```
, for example
```
cuda:sm_90
```
, to compile for a deployment target from a different host).

模型首次部署时会下载权重并编译计算图，这是最耗时的部分。两者的结果都会被缓存，因此后续部署同一模型时速度会快很多。

权重通过
```
huggingface_hub
```
下载到共享的Hugging Face缓存（默认路径为
```
~/.cache/huggingface
```
；可设置
```
HF_HOME
```
修改路径）。重新部署同一仓库会复用缓存的权重，无需重新下载。
编译结果也会被缓存。若要提前预热缓存（例如在演示或部署前），使首次
```
max serve
```
跳过下载和
```
Still compiling
```
等待，可先运行
```
max warm-cache
```
：
bash
```
max warm-cache --model <hf-repo-or-path> [--custom-architectures /abs/path/to/my_arch]
```
编译后的产物（MEF）与平台相关，因此需在同一硬件上预热缓存（或通过
```
--target
```
参数指定目标硬件，例如
```
cuda:sm_90
```
，从不同主机为部署目标编译）。

Troubleshooting

故障排查

Match the symptom against references/troubleshooting.md
. It covers the startup failures that look cryptic but have one-line fixes (encoding mismatch,

architectures[0]

name mismatch,

trust_remote_code

, OOM at load, port in use, wrong device routing). Read the serve log first; the real error is usually a few lines above the final traceback.

根据症状参考**

references/troubleshooting.md

**。该文档涵盖了看似晦涩但只需一行代码即可修复的启动失败问题（编码不匹配、

architectures[0]

名称不匹配、

trust_remote_code

serve-model

Original

Translation

Serve a model with MAX

使用MAX部署模型

References

参考文档

Fast path (custom architecture): do this first

快速流程（自定义架构）：优先执行以下步骤

1. Get the recommended command + notes (reads arch.py + config.json for you).

1. 获取推荐命令及说明（自动读取arch.py和config.json）。

2. Launch it (add
`pixi run`
if in a pixi project). On a REMOTE box, wrap with

2. 启动服务器（若在pixi项目中，需添加
`pixi run`
前缀）。在远程服务器上，使用

`setsid ... </dev/null`
so it survives the SSH session:

`setsid ... </dev/null`
确保SSH会话关闭后服务器仍能运行：

3. Wait for readiness in ONE call (fails fast on a crash; ~10 min budget for

3. 一键等待服务器就绪（启动失败时快速报错；冷编译最多等待约10分钟，出现
`Still compiling`
心跳信息为正常现象）：

a cold compile, advancing
`Still compiling`
heartbeats are normal):

4. Confirm with one request (model field must equal --served-model-name):

4. 发送请求验证（model字段必须与--served-model-name一致）：

1. Make sure MAX is installed

1. 确保已安装MAX

pixi (conda channels)

pixi（conda源）

uv (pip wheels)

uv（pip wheels）

2. Is this a built-in model or a custom architecture?

2. 判断是内置模型还是自定义架构？

3. Target a custom architecture

3. 指定自定义架构

4. Build the serve command (default-first)

4. 构建部署命令（优先使用默认配置）

5. Launch and confirm it works

5. 启动并验证服务器正常运行

Cache weights and compilation for faster re-serves

缓存权重和编译结果以加快后续部署速度

Troubleshooting

故障排查

serve-model

Original

Translation

Serve a model with MAX

使用MAX部署模型

References

参考文档

Fast path (custom architecture): do this first

快速流程（自定义架构）：优先执行以下步骤

1. Get the recommended command + notes (reads arch.py + config.json for you).

1. 获取推荐命令及说明（自动读取arch.py和config.json）。

2. Launch it (add pixi run if in a pixi project). On a REMOTE box, wrap with

2. 启动服务器（若在pixi项目中，需添加pixi run前缀）。在远程服务器上，使用

setsid ... </dev/null so it survives the SSH session:

setsid ... </dev/null 确保SSH会话关闭后服务器仍能运行：

3. Wait for readiness in ONE call (fails fast on a crash; ~10 min budget for

3. 一键等待服务器就绪（启动失败时快速报错；冷编译最多等待约10分钟，出现Still compiling心跳信息为正常现象）：

a cold compile, advancing Still compiling heartbeats are normal):

4. Confirm with one request (model field must equal --served-model-name):

4. 发送请求验证（model字段必须与--served-model-name一致）：

1. Make sure MAX is installed

1. 确保已安装MAX

pixi (conda channels)

pixi（conda源）

uv (pip wheels)

uv（pip wheels）

2. Is this a built-in model or a custom architecture?

2. 判断是内置模型还是自定义架构？

3. Target a custom architecture

3. 指定自定义架构

4. Build the serve command (default-first)

4. 构建部署命令（优先使用默认配置）

5. Launch and confirm it works

5. 启动并验证服务器正常运行

Cache weights and compilation for faster re-serves

缓存权重和编译结果以加快后续部署速度

Troubleshooting

故障排查

2. Launch it (add
`pixi run`
if in a pixi project). On a REMOTE box, wrap with

2. 启动服务器（若在pixi项目中，需添加
`pixi run`
前缀）。在远程服务器上，使用

`setsid ... </dev/null`
so it survives the SSH session:

`setsid ... </dev/null`
确保SSH会话关闭后服务器仍能运行：

3. 一键等待服务器就绪（启动失败时快速报错；冷编译最多等待约10分钟，出现
`Still compiling`
心跳信息为正常现象）：

a cold compile, advancing
`Still compiling`
heartbeats are normal):