benchmark-model

Compare original and translation side by side

🇺🇸

Original

English

🇨🇳

Translation

Chinese

Benchmark a model on MAX

在MAX上对模型进行基准测试

max benchmark

measures a running model server. It's a load generator: it sends inference requests to a live

max serve

endpoint, times them, and reports throughput (tokens/sec) and latency (TTFT, TPOT, inter-token latency). A benchmark combines two things: a server under test, and a workload that matches the question you're asking.

Every run reports both throughput and latency, so there's no mode to select. Decide what you want to learn first, then pick the workload that measures it. Single-stream latency and peak throughput come from different workloads, so the workload you choose is the measurement. Get that right and a clean number falls out.

Use this skill when you want a performance number for a model on MAX: tokens/sec, TTFT / TPOT, a concurrency or request-rate sweep, a latency-vs-throughput tradeoff, or deployment sizing.

Don't use this skill when no server is running yet.

max benchmark

is a client, so start a server with the

serve-model

skill first. To find out where inference time goes at the kernel level, use

profile-model

. To check whether the output is correct, treat it as a parity task (

import-model

, then

debug-model

) rather than a benchmark.

This skill works anywhere MAX is installed (pip or pixi). Add

pixi run

in a pixi project.

max benchmark

用于测量运行中的模型服务器。它是一个负载生成器：向实时

max serve

端点发送推理请求，计时并报告吞吐量（tokens/秒）和延迟（TTFT、TPOT、令牌间延迟）。基准测试包含两部分：被测服务器，以及与你要解决的问题匹配的工作负载。

每次运行都会同时报告吞吐量和延迟，因此无需选择模式。先确定你想了解的内容，再选择对应的测量工作负载。单流延迟和峰值吞吐量来自不同的工作负载，因此你选择的工作负载就是测量方式。选对工作负载，就能得到准确的数值。

当你需要获取MAX上模型的性能数值时，可使用此技能：每秒令牌数、TTFT/TPOT、并发或请求率扫描、延迟与吞吐量的权衡，或部署规模规划。

如果尚未启动服务器，请勿使用此技能。

max benchmark

是客户端，因此需先使用

serve-model

技能启动服务器。若要从内核层面了解推理时间的去向，请使用

profile-model

。若要检查输出是否正确，请将其视为一致性任务（

import-model

，然后

debug-model

），而非基准测试。

此技能可在任何安装了MAX的环境中使用（pip或pixi）。在pixi项目中添加

pixi run

。

References

参考资料

The following table lists the reference files and when to read each one:

File	Read when
references/flags.md	Choosing any flag or dataset beyond the ones below
references/metrics.md	Turning the throughput and latency numbers into a conclusion
references/troubleshooting.md	A run won't connect, requests fail, or the numbers look wrong

Read the reference for what you're doing, not all of them upfront.

下表列出了参考文件及其适用场景：

文件	适用场景
references/flags.md	选择以下未提及的任何标志或数据集时
references/metrics.md	将吞吐量和延迟数值转化为结论时
references/troubleshooting.md	运行无法连接、请求失败或数值异常时

按需阅读对应参考资料，无需预先通读全部内容。

1. Check the server and read its model name

1. 检查服务器并读取其模型名称

bash

curl -s http://localhost:8000/v1/health     # 200 = ready
curl -s http://localhost:8000/v1/models      # note the served model name

Both checks matter before you spend a run:

The benchmark's
```
--model
```
must equal the server's
```
--served-model-name
```
exactly, or every request fails. Take the value from
```
/v1/models
```
rather than guessing it.
If that served name is an alias rather than a Hugging Face ID (no
```
/
```
in it),
```
--tokenizer
```
defaults to it and can't resolve, and the run dies with "not a valid model identifier." Pass the model's real Hugging Face ID as
```
--tokenizer
```
.

If nothing is serving, start a server with

max serve

(for a custom architecture, use the

serve-model

skill). One more server setting matters:

--max-batch-size

caps real concurrency. A sweep to

--max-concurrency 32

against a server started with

--max-batch-size 1

queues requests instead of batching them, so raise the server's batch size to match the sweep or the high-concurrency points mean nothing.

Wait for

Server ready

then benchmark. Benchmarking during compile or warmup produces garbage first-token times.

bash

curl -s http://localhost:8000/v1/health     # 200 = 就绪
curl -s http://localhost:8000/v1/models      # 记录部署的模型名称

在开始运行前，这两项检查都至关重要：

基准测试的
```
--model
```
参数必须与服务器的
```
--served-model-name
```
完全一致，否则所有请求都会失败。请从
```
/v1/models
```
获取该值，而非猜测。
如果部署的名称是别名而非Hugging Face ID（名称中不含
```
/
```
），
```
--tokenizer
```
会默认使用该别名，但无法解析，导致运行失败并提示“不是有效的模型标识符”。此时需传入模型真实的Hugging Face ID作为
```
--tokenizer
```
参数。

如果没有服务器在运行，请使用

max serve

启动服务器（对于自定义架构，请使用

serve-model

技能）。还有一项服务器设置很重要：

--max-batch-size

会限制实际并发量。若针对以

--max-batch-size 1

启动的服务器执行

--max-concurrency 32

扫描，请求会被排队而非批量处理，因此请将服务器的批量大小调整为与扫描值匹配，否则高并发点的数值毫无意义。

等待服务器显示

Server ready

后再进行基准测试。在编译或预热阶段进行基准测试会导致首令牌时间数值无效。

2. Pick the workload for your question

2. 根据你的问题选择工作负载

The workload is the measurement. Match it to what you want to learn:

What you want to know	Workload
Best-case single-request latency (TTFT, TPOT)	`--max-concurrency 1` , fixed `random` shape, small `--num-prompts`
Peak throughput and where latency degrades	`--max-concurrency 1,2,4,8,16,32` sweep, more prompts
Performance under a realistic mix	`--dataset-name sharegpt` , moderate concurrency
Behavior at a target load	`--request-rate 1,2,4,8` sweep (requests/sec)

A sweep answers the first two rows at once: the concurrency-1 point is the best-case latency number, and the peak across the sweep is the throughput number. Reach for a dedicated concurrency-1 run when you only want the latency figure and don't want to pay for the rest of the curve.

The key knobs are the following (

references/flags.md

has the full catalog):

```
--dataset-name
```
: pick
```
random
```
(synthetic, shape it with
```
--random-input-len
```
and
```
--random-output-len
```
),
```
sharegpt
```
(real chat), or
```
arxiv-summarization
```
(long context).
```
random
```
works best for clean, reproducible micro-measurements.
```
--max-concurrency
```
and
```
--request-rate
```
: take a single value or a comma-separated sweep (
```
1,2,4,8
```
). A sweep is how you find the throughput knee.
```
--endpoint
```
: use
```
/v1/completions
```
for base LMs, which need no chat template, or
```
/v1/chat/completions
```
for instruct and chat models, which must have a chat template or the requests return 400.
```
--max-output-len
```
: sets the decode length, which dominates how long the run takes.
```
--num-prompts
```
: required for single-turn runs.

工作负载即测量方式，请根据你想了解的内容选择：

你想了解的内容	工作负载
最佳单请求延迟（TTFT、TPOT）	`--max-concurrency 1` ，固定 `random` 形状，较小的 `--num-prompts`
峰值吞吐量及延迟劣化点	`--max-concurrency 1,2,4,8,16,32` 扫描，更多提示语
真实混合场景下的性能	`--dataset-name sharegpt` ，中等并发量
目标负载下的行为	`--request-rate 1,2,4,8` 扫描（请求/秒）

一次扫描可同时回答前两行的问题：并发量为1的点是最佳延迟数值，扫描中的峰值是吞吐量数值。当你只需要延迟数值且不想为其余曲线付出成本时，可选择专门的并发量为1的运行。

关键参数如下（完整参数列表请见

references/flags.md

）：

```
--dataset-name
```
：可选
```
random
```
（合成数据，通过
```
--random-input-len
```
和
```
--random-output-len
```
调整形状）、
```
sharegpt
```
（真实对话）或
```
arxiv-summarization
```
（长上下文）。
```
random
```
最适合用于清晰、可复现的微观测量。
```
--max-concurrency
```
和
```
--request-rate
```
：可设置单个值或逗号分隔的扫描值（如
```
1,2,4,8
```
）。扫描是找到吞吐量拐点的方法。
```
--endpoint
```
：基础语言模型使用
```
/v1/completions
```
（无需聊天模板），指令型和对话型模型使用
```
/v1/chat/completions
```
（必须有聊天模板，否则请求会返回400错误）。
```
--max-output-len
```
：设置解码长度，这是决定运行时长的主要因素。
```
--num-prompts
```
：单轮运行必填参数。

3. Run it, save results, add GPU stats

3. 运行测试、保存结果并添加GPU统计信息

For best-case single-request latency, pin concurrency to 1 and keep the shape fixed:

bash

pixi run max benchmark --backend modular --base-url http://localhost:8000 \
  --model <served-model-name> --endpoint /v1/completions \
  --dataset-name random --random-input-len 128 --random-output-len 128 \
  --max-output-len 128 --num-prompts 32 --max-concurrency 1 \
  --result-filename results/latency.json --collect-gpu-stats

For peak throughput and the latency knee, sweep concurrency and send more prompts:

bash

pixi run max benchmark --backend modular --base-url http://localhost:8000 \
  --model <served-model-name> --endpoint /v1/completions \
  --dataset-name random --random-input-len 512 --random-output-len 128 \
  --max-output-len 128 --num-prompts 200 \
  --max-concurrency 1,2,4,8,16,32 \
  --result-filename results/throughput.json --collect-gpu-stats

Note these three things about saving and instrumenting a run:

```
--result-filename
```
: writes metrics to JSON and creates the directories it needs. Set it whenever you want to track or compare runs; without it, MAX saves nothing.
```
--metadata key=value
```
stamps the JSON, for example
```
--metadata tp=1 gpu=b200
```
. A sweep also drops a
```
results-<N>-median.json
```
per step under
```
--log-dir
```
.
```
--collect-gpu-stats
```
: adds GPU utilization and peak memory. This works only when the benchmark runs on the same machine as the server (NVIDIA).
For version-controlled configs, put options under a
```
benchmark_config:
```
key in a YAML file and pass
```
--config-file file.yaml
```
. Keys use
```
snake_case
```
, and CLI flags override the file.

如需获取最佳单请求延迟，请将并发量固定为1并保持形状不变：

bash

pixi run max benchmark --backend modular --base-url http://localhost:8000 \
  --model <served-model-name> --endpoint /v1/completions \
  --dataset-name random --random-input-len 128 --random-output-len 128 \
  --max-output-len 128 --num-prompts 32 --max-concurrency 1 \
  --result-filename results/latency.json --collect-gpu-stats

如需获取峰值吞吐量和延迟拐点，请扫描并发量并发送更多提示语：

bash

pixi run max benchmark --backend modular --base-url http://localhost:8000 \
  --model <served-model-name> --endpoint /v1/completions \
  --dataset-name random --random-input-len 512 --random-output-len 128 \
  --max-output-len 128 --num-prompts 200 \
  --max-concurrency 1,2,4,8,16,32 \
  --result-filename results/throughput.json --collect-gpu-stats

关于保存和监控运行，请注意以下三点：

```
--result-filename
```
：将指标写入JSON文件，并自动创建所需目录。当你需要跟踪或对比运行结果时请设置该参数；若未设置，MAX不会保存任何内容。
```
--metadata key=value
```
可为JSON文件添加标记，例如
```
--metadata tp=1 gpu=b200
```
。扫描还会在
```
--log-dir
```
下为每个步骤生成
```
results-<N>-median.json
```
文件。
```
--collect-gpu-stats
```
：添加GPU利用率和峰值内存统计。此功能仅在基准测试与服务器运行在同一台机器（NVIDIA）上时生效。
对于版本控制的配置，可在YAML文件的
```
benchmark_config:
```
键下设置参数，然后通过
```
--config-file file.yaml
```
传入。键名使用
```
snake_case
```
，CLI标志会覆盖文件中的设置。

4. Read the metrics

4. 解读指标

The run prints throughput and latency, and a sweep prints one row per point. The headline numbers are the following:

Output token throughput (tok/s): the main throughput number.
TTFT (time to first token): prefill responsiveness. Watch p50 and p99.
TPOT and ITL (time per output token and inter-token latency): decode speed.
GPU utilization and peak memory: reported with
```
--collect-gpu-stats
```
.

For how to turn these numbers into a conclusion, and the latency-vs-throughput tradeoff a sweep reveals, see

references/metrics.md

运行会打印吞吐量和延迟，扫描会为每个点打印一行数据。核心指标如下：

输出令牌吞吐量（tok/s）：主要的吞吐量数值。
TTFT（首令牌时间）：预填充响应速度。请关注p50和p99数值。
TPOT和ITL（每个输出令牌的时间和令牌间延迟）：解码速度。
GPU利用率和峰值内存：启用
```
--collect-gpu-stats
```
后会报告这些指标。

如需了解如何将这些数值转化为结论，以及扫描揭示的延迟与吞吐量权衡关系，请查看

references/metrics.md

。

Troubleshooting

故障排除

Match the symptom against

references/troubleshooting.md

, which covers connection failures, model-name mismatches, tokenizer-alias errors, chat-template 400s, flat throughput from a batch-size cap, and warmup-skewed first-token times. Confirm that

curl /v1/health

returns 200 before you check anything else.

请根据症状参考

references/troubleshooting.md

，其中涵盖了连接失败、模型名称不匹配、分词器别名错误、聊天模板导致的400错误、批量大小限制导致的吞吐量停滞，以及预热阶段导致的首令牌时间异常等问题。在检查其他内容之前，请先确认

curl /v1/health

返回200。