debug-model

Compare original and translation side by side

🇺🇸

Original

English

🇨🇳

Translation

Chinese

Parity/coherence failure protocol

一致性/连贯性失败排查流程

The model runs without errors but output is wrong. Scalar

ops.print

taps and recompile loops hide directional bugs and burn GPU time. Build a per-layer tensor-dump comparator first; every later check becomes a numpy read from disk.

Use this skill when MAX output disagrees with a PyTorch reference you can run and hook. The primary case is a custom-architecture port that serves but fails parity or coherence checks; the same protocol covers a quantized variant of a working port, a multi-GPU conversion of a working single-GPU port, and a regression after a MAX upgrade — anywhere a trusted reference exists.

Do not use this skill when:

The server crashes on load → fix config, weights, graph (
```
import-model
```
)
You have not finished implementing the graph →
```
import-model
```
Phase 2
An already-verified model needs logit-comparison tolerances tuned → that is threshold calibration, not corruption

模型运行无报错但输出结果错误。标量

ops.print

探测和重新编译循环会掩盖方向性bug并消耗GPU时间。应先构建逐层张量转储比较器；后续所有检查都将基于磁盘上的numpy读取结果。

适用场景：当MAX输出与你可运行并挂钩的PyTorch参考实现不一致时。主要场景为自定义架构移植完成可服务但未通过一致性或连贯性检查；该流程同样适用于已验证移植版本的量化变体、单GPU移植版本的多GPU转换，以及MAX版本升级后的回归问题——任何存在可信参考实现的场景均可使用。

不适用场景：

服务器加载时崩溃 → 修复配置、权重、计算图（使用
```
import-model
```
）
计算图尚未实现完成 → 进入
```
import-model
```
第二阶段
已验证模型需要调整logit比较容差 → 这属于阈值校准，并非损坏问题

References

参考文档

File	Read when
comparator-build.md	Building HF/MAX dumpers and the comparator
comparator-output-patterns.md	Interpreting comparator output, false cliffs, token-0 invariant
agent-workflow.md	Dispatching parallel investigation agents
stacked-failures.md	A fix helped but verification still fails

For MAX's built-in runtime debugging options (NaN checks, source tracebacks, op logging), see the MAX debugging tools.

max.nn.hooks.PrintHook

(covered in comparator-build.md) prints layer inputs and outputs for quick triage.

文件链接	阅读时机
comparator-build.md	构建HF/MAX转储工具及比较器时
comparator-output-patterns.md	解读比较器输出、排查假 cliff、token-0不变量时
agent-workflow.md	调度并行调查Agent时
stacked-failures.md	修复生效但验证仍未通过时

关于MAX内置的运行时调试选项（NaN检查、源码回溯、操作日志），请查看 MAX调试工具文档。

max.nn.hooks.PrintHook

（在comparator-build.md中介绍）可打印层的输入和输出，用于快速分类排查。

Protocol

排查流程

Step 0: Sanity-check HF

步骤0：HF sanity检查

Run

model.generate(...)

on the same HF repo, prompt, and checkpoint. If HF is incoherent, fix tokenizer/chat-template first; the MAX graph is not the problem.

bash

pixi run python -c "
import torch
from transformers import AutoTokenizer, AutoModelForCausalLM
tok = AutoTokenizer.from_pretrained('<repo>')
model = AutoModelForCausalLM.from_pretrained('<repo>', torch_dtype=torch.bfloat16, device_map='auto')
text = tok.apply_chat_template([{'role':'user','content':'Hello!'}], tokenize=False, add_generation_prompt=True)
out = model.generate(**tok(text, return_tensors='pt').to(model.device), max_new_tokens=32, do_sample=False)
print(tok.decode(out[0]))
"

在相同的HF仓库、提示词和检查点上运行

model.generate(...)

。如果HF输出不连贯，先修复tokenizer/聊天模板；此时MAX计算图并非问题所在。

bash

pixi run python -c "
import torch
from transformers import AutoTokenizer, AutoModelForCausalLM
tok = AutoTokenizer.from_pretrained('<repo>')
model = AutoModelForCausalLM.from_pretrained('<repo>', torch_dtype=torch.bfloat16, device_map='auto')
text = tok.apply_chat_template([{'role':'user','content':'Hello!'}], tokenize=False, add_generation_prompt=True)
out = model.generate(**tok(text, return_tensors='pt').to(model.device), max_new_tokens=32, do_sample=False)
print(tok.decode(out[0]))
"

Step 1: Build the comparator

步骤1：构建比较器

Follow comparator-build.md. You need three artifacts: HF dumper, MAX dumper (graph edits + standalone runner), comparator script. Cast dump tensors to FP32 in the MAX graph.

Guard: validate the dumpers before trusting them. Run both dumpers on a model MAX already serves correctly (any registered Llama works). Expect cos ≈ 0.999 at every layer, identical
prompt_tokens.npy
on both sides, and
post_embed
cos = 1.0. Anything less means the dumpers are broken — fix them before reading anything into a comparison on your port.

遵循comparator-build.md的指引。你需要三个产物：HF转储工具、MAX转储工具（计算图修改 + 独立运行器）、比较器脚本。在MAX计算图中将转储张量转换为FP32格式。

注意：在信任转储工具前先验证其正确性。 在MAX已正常服务的模型（任何已注册的Llama模型均可）上运行两个转储工具。预期每一层的余弦相似度≈0.999，两侧的
prompt_tokens.npy
完全一致，且
post_embed
的余弦相似度=1.0。若未达到该标准，说明转储工具存在问题——在对你的移植版本进行比较前，先修复转储工具。

Step 2: Read comparator output, then branch

步骤2：解读比较器输出，选择分支

Follow comparator-output-patterns.md. Check false cliffs (wrong

hidden_states

indexing, missing

attention_mask

on decode-prefix dumps) before bisecting the graph.

The first trustworthy comparator run is a fork, not a checkpoint:

Some layer diverges → graph hunt; continue with Steps 3 to 5.
Every layer matches (cos ≥ 0.99) but generation still diverges → the graph is likely correct. Skip to Step 6; do not bisect layers.
Pattern matches a false-cliff signature → fix the dumper, re-dump, re-read. Do not debug the graph against a broken comparator.

Compute per-token and per-dim cosine slices when global cos looks ambiguous:

python

cos_per_token = [cos(h[t], m[t]) for t in range(h.shape[0])]
cos_per_dim   = [cos(h[:,d], m[:,d]) for d in range(h.shape[1])]

High

max_diff

where HF spikes and MAX is flat usually means HF formed an attention anchor your port did not, not "MAX exploding."

遵循comparator-output-patterns.md的指引。在对计算图进行二分排查前，先检查假cliff（错误的

hidden_states

索引、解码前缀转储时缺失

attention_mask

）问题。

首次可信的比较器运行结果是分支依据，而非检查点：

某一层出现分歧 → 排查计算图；继续执行步骤3至5。
每一层都匹配（余弦相似度≥0.99）但生成结果仍分歧 → 计算图大概率是正确的。跳至步骤6；无需对层进行二分排查。
输出模式匹配假cliff特征 → 修复转储工具，重新转储并解读。不要基于损坏的比较器调试计算图。

当全局余弦相似度模糊时，计算逐token和逐维度的余弦切片：

python

cos_per_token = [cos(h[t], m[t]) for t in range(h.shape[0])]
cos_per_dim   = [cos(h[:,d], m[:,d]) for d in range(h.shape[1])]

当HF出现峰值而MAX保持平稳时的高

max_diff

，通常意味着HF形成了你的移植版本未实现的注意力锚点，而非「MAX出现异常激增」。

Step 3: Dispatch investigation agents

步骤3：调度调查Agent

Follow agent-workflow.md. One lead agent analyzes dumps and ranks hypotheses with tensor evidence. Helpers run in parallel (weight stats, code diff, kernel inspection, sub-tap prep). Do not dispatch fix-attempt agents until the lead localizes.

遵循agent-workflow.md的指引。一名主导Agent分析转储结果并结合张量证据对假设进行排序。辅助Agent并行执行任务（权重统计、代码差异、内核检查、子探测准备）。在主导Agent定位问题前，不要调度尝试修复的Agent。

Step 4: Verify numerically before recompiling

步骤4：重新编译前先进行数值验证

For each hypothesis: read dump tensors, compute what the fix would produce, compare to HF. Match → recompile. No match → next hypothesis.

针对每个假设：读取转储张量，计算修复后的预期结果，并与HF进行比较。匹配则重新编译。不匹配则尝试下一个假设。

Step 5: Apply fix, re-dump, re-compare

步骤5：应用修复，重新转储并比较

One compile, one smoke, full comparator pass (cos > 0.99 all layers). If verification still fails, see stacked-failures.md.

编译一次，进行一次快速测试，然后完成全量比较器检查（所有层余弦相似度>0.99）。若验证仍未通过，请查看stacked-failures.md。

Step 6: Serve vs pipeline

步骤6：服务端与流水线对比

When teacher-forced dumps at decode step K match HF but generated text diverges, the graph is likely correct. Bisect before re-bisecting layers:

Check	Pass	Fail →
Teacher-forced dump @ K	cos ≥ 0.99, argmax matches	Steps 1 to 5 (graph bug)
Incremental pipeline decode @ K	token K matches HF	Decode-state bug (KV, conv cache)
Serve vs pipeline @ K	match	Harness bug (tokenizer, chat template, token recovery)

Build if missing: pipeline decode compare, incremental layer dump, serve compare scripts. If teacher-forced and pipeline both pass but serve fails, do not edit the graph.

当解码步骤K的强制教师转储与HF匹配但生成文本仍分歧时，计算图大概率是正确的。在重新对层进行二分排查前，先执行以下二分检查：

检查项	通过标准	失败 → 排查方向
步骤K的强制教师转储	余弦相似度≥0.99，argmax匹配	执行步骤1至5（计算图bug）
步骤K的增量流水线解码	token K与HF匹配	解码状态bug（KV缓存、卷积缓存）
步骤K的服务端与流水线对比	结果匹配	测试框架bug（tokenizer、聊天模板、token恢复逻辑）

若缺少相关工具则构建：流水线解码对比工具、增量层转储工具、服务端对比脚本。若强制教师转储和流水线均通过但服务端失败，请勿修改计算图。