magpie-kernel-evaluator

Compare original and translation side by side

🇺🇸

Original

English

🇨🇳

Translation

Chinese

Magpie

Use Magpie for three connected jobs:

Benchmark an inference workload and collect throughput, latency, and traces.
Analyze or compare GPU kernels for correctness and performance.
Drive an optimization loop from a benchmark bottleneck to source, candidate kernels, and end-to-end validation.

Describe only capabilities supported by the checked-out Magpie version. Do not infer support for an unverified ROCm, GPU, framework, or experimental integration.

Magpie可用于完成三项关联任务：

对推理工作负载进行基准测试，收集吞吐量、延迟和轨迹数据。
分析或对比GPU内核的正确性与性能。
推动从基准测试瓶颈到源代码、候选内核再到端到端验证的优化循环。

仅描述当前检出的Magpie版本所支持的功能。请勿推断未经验证的ROCm、GPU、框架或实验性集成的支持情况。

Choose the workflow

选择工作流

User goal	Workflow
Evaluate one implementation	`analyze`
Rank two or more implementations	`compare`
Measure model-serving performance	`benchmark`
Find expensive kernels in existing traces	standalone gap analysis
Explain a profiled inference workload	benchmark → TraceLens post-processing → stage/roofline review
Optimize an end-to-end workload	benchmark → TraceLens/gap analysis → source mapping → analyze/compare → re-benchmark

Use a YAML config for reproducible or multi-step work. Use inline CLI arguments for small exploratory runs.

用户目标	工作流
评估单个实现	`analyze`
对两个或多个实现进行排序	`compare`
衡量模型服务性能	`benchmark`
在现有轨迹中查找高开销内核	独立差距分析
解释已分析的推理工作负载	benchmark → TraceLens后处理 → 阶段/roofline模型审查
优化端到端工作负载	benchmark → TraceLens/差距分析 → 源代码映射 → analyze/compare → 重新基准测试

对于可复现或多步骤工作，建议使用YAML配置。小型探索性运行可使用内联CLI参数。

Preflight

预检查

Locate the Magpie repository or installed package.

Check the local interface before constructing commands:

bash

magpie --help
magpie analyze --help
magpie compare --help
magpie benchmark --help
magpie --gpu-info

Check required tools, model access, GPU visibility, writable output space, and container or Ray access as applicable.
Read the repository compatibility matrix before making version claims. Treat ROCm or hardware not listed there as unverified until tested.
Record the exact config, model revision, image, environment variables, GPU allocation, and Magpie commit for benchmark comparisons.

Run from the Magpie repository root, install with

pip install -e .

, or use

python -m Magpie

when the

magpie

entry point is unavailable.

找到Magpie仓库或已安装的包。

在构建命令前检查本地接口：

bash

magpie --help
magpie analyze --help
magpie compare --help
magpie benchmark --help
magpie --gpu-info

检查所需工具、模型访问权限、GPU可见性、可写输出空间，以及容器或Ray访问权限（如适用）。
在声明版本兼容性前，阅读仓库的兼容性矩阵。未在其中列出的ROCm或硬件在测试前均视为未经验证。
记录基准测试对比所需的精确配置、模型版本、镜像、环境变量、GPU分配和Magpie提交记录。

从Magpie仓库根目录运行，使用

pip install -e .

安装，或者当

magpie

入口不可用时，使用

python -m Magpie

。

Analyze a kernel

分析内核

Prefer a config when correctness or profiler settings matter:

bash

magpie analyze --kernel-config path/to/kernel.yaml

For a quick single-kernel run:

bash

magpie analyze path/to/kernel.hip --type hip --testcase "./run_test.sh"

Supported public kernel types are

hip

cuda

pytorch

, and

triton

. Use

--no-perf

only when the user wants correctness or execution validation without profiling.

Do not equate successful execution with numerical correctness. Supply a representative testcase whenever an optimized result will be accepted or rejected.

当正确性或分析器设置很重要时，优先使用配置文件：

bash

magpie analyze --kernel-config path/to/kernel.yaml

快速运行单个内核：

bash

magpie analyze path/to/kernel.hip --type hip --testcase "./run_test.sh"

支持的公开内核类型包括

hip

、

cuda

、

pytorch

和

triton

。仅当用户希望在不进行性能分析的情况下验证正确性或执行情况时，使用

--no-perf

参数。

请勿将执行成功等同于数值正确性。每当需要接受或拒绝优化结果时，都要提供具有代表性的测试用例。

Compare kernel variants

对比内核变体

Compare at least two implementations and identify the baseline explicitly:

bash

magpie compare --kernel-config path/to/compare.yaml

Keep inputs, tolerances, warmup, iteration count, GPU allocation, and profiler settings identical across candidates. Reject candidates that fail correctness before considering performance rankings.

For PyTorch without a testcase, Magpie's built-in check only verifies that each result is finite; it does not prove numerical equivalence between variants. Require a testcase for numerical validation.

至少对比两个实现，并明确指定基准版本：

bash

magpie compare --kernel-config path/to/compare.yaml

确保所有候选内核的输入、容差、预热次数、迭代次数、GPU分配和分析器设置完全一致。在考虑性能排名前，先排除正确性测试未通过的候选内核。

对于没有测试用例的PyTorch，Magpie的内置检查仅验证每个结果是否为有限值，并不证明变体之间的数值等价性。如需数值验证，必须提供测试用例。

Benchmark inference

推理基准测试

Prefer a checked-in benchmark config:

bash

magpie benchmark --benchmark-config path/to/benchmark.yaml

The stable public CLI supports

vllm

sglang

, and

atom

. It supports direct

docker

and

local

run modes; use YAML configuration and the repository's Ray examples for distributed execution. Do not advertise integrations that exist only in internal enums or partial code paths as stable.

Enable profiling deliberately: profiler runs perturb latency and should not replace a clean baseline. Compare throughput, completed requests, TTFT, TPOT, ITL, and end-to-end latency using equivalent workloads.

优先使用已提交的基准测试配置：

bash

magpie benchmark --benchmark-config path/to/benchmark.yaml

稳定的公开CLI支持

vllm

、

sglang

和

atom

。支持直接的

docker

和

local

运行模式；分布式执行需使用YAML配置和仓库中的Ray示例。请勿将仅存在于内部枚举或部分代码路径中的集成宣传为稳定功能。

谨慎启用性能分析：性能分析运行会干扰延迟数据，不能替代干净的基准测试。使用等效工作负载对比吞吐量、完成请求数、TTFT、TPOT、ITL和端到端延迟。

Post-process traces with TraceLens

使用TraceLens后处理轨迹

Enable TraceLens in the profiled benchmark YAML; torch traces are its required input:

yaml

benchmark:
  profiler:
    torch_profiler:
      enabled: true
    tracelens:
      enabled: true
      analysis_mode: inference
      analysis_stages: all
      export_format: csv

Use

analysis_mode: inference

for vLLM/SGLang. It splits the rank-0 trace into

prefilldecode

decode

, and

prefill

stages when available, runs TraceLens post-processing, and writes full stage reports plus compact

*_kernel_roofline_simple.csv

files under the benchmark workspace's

tracelens/

directory. For direct PyTorch trace reporting, use

analysis_mode: pytorch

Open the compact roofline CSVs first. Rank rows by

kernel_time_ms_sum

time_pct

; then use

roofline_bound

, arithmetic intensity, achieved TFLOP/s or TB/s, and

pct_roofline_mean

to form an optimization hypothesis. Confirm

benchmark_report.json.tracelens_analysis

has outputs and no error before treating post-processing as successful. Use

analysis_mode: pytorch

when the task specifically needs the legacy direct single-rank or multi-rank collective reports.

Magpie's integrated TraceLens stage produces CSV/Excel analysis artifacts, not an agent-written

analysis.md

. If the user requests a prioritized agentic report, pass the captured trace to the separate

tracelens-analysis-orchestrator

skill when installed; keep that result distinct from Magpie's benchmark report.

在已开启性能分析的基准测试YAML中启用TraceLens；torch轨迹是其必需输入：

yaml

benchmark:
  profiler:
    torch_profiler:
      enabled: true
    tracelens:
      enabled: true
      analysis_mode: inference
      analysis_stages: all
      export_format: csv

对vLLM/SGLang使用

analysis_mode: inference

。如果可用，它会将rank-0轨迹拆分为

prefilldecode

、

decode

和

prefill

阶段，运行TraceLens后处理，并在基准测试工作区的

tracelens/

目录下生成完整的阶段报告以及精简的

*_kernel_roofline_simple.csv

文件。对于直接的PyTorch轨迹报告，使用

analysis_mode: pytorch

。

优先打开精简的roofline CSV文件。按

kernel_time_ms_sum

或

time_pct

对行排序；然后使用

roofline_bound

、算术强度、已达到的TFLOP/s或TB/s以及

pct_roofline_mean

形成优化假设。在将后处理视为成功前，确认

benchmark_report.json.tracelens_analysis

有输出且无错误。当任务特别需要传统的直接单rank或多rank集合报告时，使用

analysis_mode: pytorch

。

Magpie集成的TraceLens阶段会生成CSV/Excel分析产物，而非由Agent编写的

analysis.md

。如果用户需要优先级排序的Agent报告，在安装了独立的

tracelens-analysis-orchestrator

技能后，将捕获的轨迹传入该技能；并将其结果与Magpie的基准测试报告区分开。

Analyze existing traces and find source

分析现有轨迹并查找源代码

Run standalone gap analysis with

--trace-dir

directly on

benchmark

bash

magpie benchmark \
  --trace-dir path/to/torch_trace \
  --top-k 20 \
  --find-kernel-sources \
  --kernel-source-repos path/to/repository

Do not insert a

gap-analysis

positional token; it is not a CLI subcommand. Inspect the generated aggregate and per-rank CSVs, and preserve source-mapping confidence rather than assuming every normalized kernel name maps uniquely.

直接在

benchmark

上使用

--trace-dir

运行独立差距分析：

bash

magpie benchmark \
  --trace-dir path/to/torch_trace \
  --top-k 20 \
  --find-kernel-sources \
  --kernel-source-repos path/to/repository

请勿插入

gap-analysis

位置参数；它不是CLI子命令。检查生成的聚合和每个rank的CSV文件，并保留源代码映射的可信度，不要假设每个标准化内核名称都能唯一映射。

Drive the optimization loop

推动优化循环

Run an unprofiled baseline benchmark and save its config and report.
Repeat with torch profiling and TraceLens inference post-processing enabled.
Review stage-level TraceLens roofline summaries to classify dominant operations and likely compute, memory, or communication limits.
Run gap analysis over the representative steady-state window to rank concrete kernels.
Select bottlenecks by total contribution, not only single-dispatch duration.
Map the selected kernel to source and an executable testcase.
Generate isolated candidate implementations; preserve the baseline.
Use
```
analyze
```
for iteration, then
```
compare
```
with correctness gates to rank candidates.
Re-run the original unprofiled benchmark with the winning candidate and the same workload. Report both kernel-level and end-to-end changes, including regressions.

Stop before claiming success if correctness is unproven, the benchmark inputs changed, the source mapping is uncertain, or the end-to-end improvement is within run-to-run noise.

运行未开启性能分析的基准测试基线，并保存其配置和报告。
开启torch性能分析和TraceLens推理后处理，重复运行。
查看TraceLens的阶段级roofline摘要，对主导操作以及可能的计算、内存或通信限制进行分类。
在代表性稳态窗口上运行差距分析，对具体内核进行排序。
按总贡献选择瓶颈，而非仅按单次调度时长。
将选定的内核映射到源代码和可执行测试用例。
生成独立的候选实现；保留基线版本。
使用
```
analyze
```
进行迭代，然后使用带有正确性检查的
```
compare
```
对候选内核排序。
使用获胜候选内核和相同工作负载重新运行原始未开启性能分析的基准测试。报告内核级和端到端的变化，包括性能退化情况。

如果正确性未得到验证、基准测试输入发生变化、源代码映射不确定或端到端改进在运行误差范围内，则请勿宣称优化成功。

Use MCP tools when available

可用时使用MCP工具

Prefer Magpie MCP tools for structured agent workflows such as hardware inspection, kernel discovery, config generation, analyze/compare, optimization suggestions, result lookup, report comparison, Ray job management, and benchmark batches.

Do not pass a CLI

analyze_report.json

wrapper directly to an MCP tool that expects one result object's

performance_state

and

performance_result

. Do not assume every CLI option exists in MCP; kernel-source enrichment is currently exposed by the CLI gap-analysis path.

对于结构化Agent工作流（如硬件检查、内核发现、配置生成、分析/对比、优化建议、结果查询、报告对比、Ray作业管理和基准测试批量处理），优先使用Magpie MCP工具。

请勿将CLI的

analyze_report.json

包装器直接传递给需要单个结果对象的

performance_state

和

performance_result

的MCP工具。请勿假设每个CLI选项在MCP中都存在；内核源代码增强目前仅通过CLI差距分析路径暴露。

Additional resources

额外资源

Full CLI reference: reference.md
Copy-paste command examples: examples.md

完整CLI参考：reference.md
可复制的命令示例：examples.md