debug-with-langwatch

Compare original and translation side by side

🇺🇸

Original

English

🇨🇳

Translation

Chinese

Debug Production Issues with LangWatch

使用 LangWatch 调试生产问题

A structured diagnostic workflow: errored traces → span inspection → monitor/evaluator scores → root cause. Work the steps in order; each narrows the search space for the next.

If traces themselves look broken (empty inputs/outputs, disconnected spans), switch to the

debug-instrumentation

recipe instead. That is an instrumentation problem, not an application problem.

一个结构化诊断工作流：出错的 trace → span 检查 → monitor/evaluator 得分 → 根因。请按顺序执行各个步骤；每一步都会为下一步缩小搜索范围。

如果 trace 本身看起来异常（输入或输出为空、span 断开连接），改用

debug-instrumentation

recipe。这是 instrumentation 问题，而不是 application 问题。

Prerequisites

前置条件

Use

langwatch docs <path>

to read documentation as Markdown. Some useful entry points:

bash

langwatch docs                                    # Docs index
langwatch docs integration/python/guide           # Python integration
langwatch docs integration/typescript/guide       # TypeScript integration
langwatch docs prompt-management/cli              # Prompts CLI
langwatch scenario-docs                           # Scenario docs index

Discover commands with

langwatch --help

and

langwatch <subcommand> --help

. List and get commands accept

--format json

for machine-readable output. Every list command takes

--limit <n>

to cap the rows and

--jq <expr>

to read part of the answer. A paginated list answers with an envelope, so count its rows through the row array (

--jq '.traces | length'

), and read how many there are in all at

.pagination.total

. Bare

--jq length

counts the fields of the envelope, not the rows. Read the docs first instead of guessing SDK APIs or CLI flags.

If no shell is available, fetch the same Markdown over plain HTTP. Append

.md

to any docs path (e.g. https://langwatch.ai/docs/integration/python/guide.md). Index: https://langwatch.ai/docs/llms.txt. Scenario index: https://langwatch.ai/scenario/llms.txt

If anything fails or confuses you while following this skill (broken commands, docs that do not match reality, errors you had to work around), ask the user for permission and run

npx --yes langwatch report --user-approved

with a

--title

and

--summary

(or

--session <transcript.jsonl>

) to send it to the LangWatch team, and it directly shapes what gets fixed. No login or API key needed. Nothing is sent without

--user-approved

, and

--dry-run

prints the exact payload without sending anything. The title, summary and transcript are scrubbed locally first, by pattern: secrets and API keys, plus email addresses, phone numbers, card numbers and public IPv4 addresses. Anything no pattern matches is sent as written, including a contact address passed with

--email

. With

--session

, always run

--dry-run

first and let the user read the payload, because a transcript carries content they never reviewed.

npx --yes langwatch report --help

explains the options.

使用

langwatch docs <path>

以 Markdown 格式阅读文档。以下是一些有用的入口：

bash

langwatch docs                                    # Docs index
langwatch docs integration/python/guide           # Python integration
langwatch docs integration/typescript/guide       # TypeScript integration
langwatch docs prompt-management/cli              # Prompts CLI
langwatch scenario-docs                           # Scenario docs index

通过

langwatch --help

和

langwatch <subcommand> --help

发现可用命令。list 和 get 命令接受

--format json

，用于输出机器可读的结果。每个 list 命令都接受

--limit <n>

来限制返回的行数，并接受

--jq <expr>

来读取结果中的指定部分。分页 list 命令返回的是一个封装结构，因此需要通过行数组统计条目数量（

--jq '.traces | length'

），而总条目数位于

.pagination.total

。单独使用

--jq length

统计的是封装结构中的字段数量，而不是行数。请先阅读文档，不要猜测 SDK API 或 CLI 参数。

如果没有可用的 shell，可以通过普通 HTTP 获取相同的 Markdown 文档。在任意文档路径后追加

.md

（例如 https://langwatch.ai/docs/integration/python/guide.md）。索引：https://langwatch.ai/docs/llms.txt。Scenario 索引：https://langwatch.ai/scenario/llms.txt

如果在执行此 skill 期间遇到任何失败或困惑（命令不可用、文档与实际情况不一致、不得不处理错误等），请征得用户许可，然后运行

npx --yes langwatch report --user-approved

，并提供

--title

和

--summary

（或

--session <transcript.jsonl>

），将问题发送给 LangWatch 团队，这会直接影响后续的修复内容。无需登录或 API key。未提供

--user-approved

时不会发送任何内容；

--dry-run

会在不发送内容的情况下打印准确的 payload。title、summary 和 transcript 会先在本地根据匹配模式进行脱敏，包括 secret、API key、email 地址、电话号码、卡号和公开 IPv4 地址。未匹配任何模式的内容会按原样发送，包括通过

--email

传入的联系地址。使用

--session

时，必须先运行

--dry-run

并让用户阅读 payload，因为 transcript 中可能包含用户从未查看过的内容。

npx --yes langwatch report --help

会说明各个选项。

Step 0: Point the CLI at the Right Project

步骤 0：让 CLI 指向正确的项目

bash

langwatch status

A fast sanity check that the API key, endpoint, and project are the ones you mean to debug. Fix auth first (see the

setup-lw

recipe): every later step reads from this project.

bash

langwatch status

这是一个快速 sanity check，用于确认 API key、endpoint 和项目确实是需要调试的目标。请先修复身份验证问题（参见

setup-lw

recipe）：后续所有步骤都会读取这个项目的数据。

Step 1: Find the Errored Traces

步骤 1：查找出错的 trace

bash

langwatch trace search --errors-only --limit 25 -o json
langwatch trace search --errors-only -q "timeout" --start-date 2026-01-01 -o json

```
--errors-only
```
is how you find failures. An error is recorded on the span, not in the trace's searchable text, so
```
-q "error"
```
finds nothing and reads like a clean project.
```
--start-date
```
/
```
--end-date
```
bound the window (ISO strings or epoch ms; default is the last 24h).
```
-q
```
does a text search over one phrase: the error message, a user id, a thread id. AND, OR and NOT are matched as words, not as operators.
The result is
```
{ "traces": [...], "pagination": { "totalHits": N } }
```
. Pull fields out with
```
--jq
```
instead of reading the whole payload:

bash

langwatch trace search --limit 50 -o json --jq ".traces[].traceId"
langwatch trace search -q "refund" -o json --jq ".traces | length"

Look for: traces with error statuses, empty or truncated outputs, outliers in latency or cost, and repeats of the same failure across users/threads (a pattern, not a one-off).

bash

langwatch trace search --errors-only --limit 25 -o json
langwatch trace search --errors-only -q "timeout" --start-date 2026-01-01 -o json

```
--errors-only
```
用于查找失败。错误记录在 span 上，而不是 trace 的可搜索文本中，因此
```
-q "error"
```
找不到任何内容，看起来会让人误以为项目没有问题。
```
--start-date
```
/
```
--end-date
```
用于限定时间范围（可使用 ISO 字符串或 epoch ms；默认值为最近 24 小时）。
```
-q
```
会对一个短语执行文本搜索，例如错误消息、user id 或 thread id。AND、OR 和 NOT 会按普通单词匹配，而不是作为运算符处理。
结果格式为
```
{ "traces": [...], "pagination": { "totalHits": N } }
```
。请使用
```
--jq
```
提取字段，而不是读取完整 payload：

bash

langwatch trace search --limit 50 -o json --jq ".traces[].traceId"
langwatch trace search -q "refund" -o json --jq ".traces | length"

重点关注：状态为错误的 trace、空或被截断的输出、延迟或成本离群值，以及不同用户或 thread 中重复出现的同类失败（这是系统性模式，而非偶发问题）。

Step 2: Inspect the Failing Spans

步骤 2：检查失败的 span

bash

langwatch trace get <traceId>            # human-readable digest
langwatch trace get <traceId> -o json    # full span hierarchy

Read the span tree top-down:

Which span failed? The error is usually in one span (an LLM call, a tool call), not the whole trace. Note its input: a bad input upstream often explains a failure downstream.
What did the model see? Check the prompt/messages on the failing LLM span. Missing context, truncated history, and stale retrieved documents are the usual suspects.
Retries and timeouts: repeated identical spans suggest retry loops; a long-running span before the failure suggests a timeout.

bash

langwatch trace get <traceId>            # human-readable digest
langwatch trace get <traceId> -o json    # full span hierarchy

自上而下阅读 span 树：

是哪个 span 失败了？ 错误通常出现在某个 span（一次 LLM 调用或 tool call）中，而不是整个 trace。记下它的输入：上游的不良输入往往可以解释下游的失败。
模型看到了什么？ 检查失败 LLM span 上的 prompt/messages。缺少上下文、历史记录被截断，以及检索到的文档过期，通常是主要嫌疑因素。
重试和超时： 重复出现完全相同的 span 表明可能存在重试循环；失败前存在长时间运行的 span，则可能意味着发生了超时。

Step 3: Check Monitors and Evaluator Scores

步骤 3：检查 monitor 和 evaluator 得分

Production quality signals live in monitors (online evaluation) and their evaluators:

bash

langwatch monitor list -o json           # which monitors exist, are they enabled/firing?
langwatch monitor get <id> -o json       # one monitor's config and recent state
langwatch evaluator list -o json         # the evaluators the monitors run

A firing monitor names the failure mode (toxicity, hallucination, PII). Corroborate it against the spans from Step 2.
No monitor for the failure mode you found? That is a gap worth closing once the root cause is fixed (
```
langwatch monitor create
```
).

For a quantitative view of the blast radius:

bash

langwatch analytics query -m trace-count -a sum --group-by metadata.model -o json

生产质量信号存在于 monitor（在线评估）及其 evaluator 中：

bash

langwatch monitor list -o json           # which monitors exist, are they enabled/firing?
langwatch monitor get <id> -o json       # one monitor's config and recent state
langwatch evaluator list -o json         # the evaluators the monitors run

已触发的 monitor 会指出具体的失败模式（toxicity、hallucination、PII）。将结果与步骤 2 中的 span 进行交叉验证。
如果没有针对所发现失败模式的 monitor，这说明仍存在监控缺口，待根因修复后应补充相应 monitor（
```
langwatch monitor create
```
）。

要定量了解问题的影响范围：

bash

langwatch analytics query -m trace-count -a sum --group-by metadata.model -o json

Step 4: Root Cause and Verify

步骤 4：定位根因并验证

Form a hypothesis from the failing span's input + the monitor's failure mode: prompt change, model change, bad retrieval, code regression.
```
git log
```
on the agent's code and prompts tells you what changed when the failures started.
Apply the fix (prompt, code, or configuration).
Generate fresh traffic, then re-run Step 1: the errored traces should stop appearing.
If the failure was a regression, add a scenario so it stays fixed. The
```
scenarios
```
skill covers this.

根据失败 span 的输入和 monitor 所反映的失败模式提出假设：prompt 变更、model 变更、检索错误或代码回归。对 Agent 的代码和 prompt 执行
```
git log
```
，即可了解故障开始时发生了哪些变化。
应用修复方案（修改 prompt、代码或配置）。
生成新的流量，然后重新执行步骤 1：出错的 trace 应不再出现。
如果故障属于回归问题，请添加 scenario，防止问题再次出现。
```
scenarios
```
skill 介绍了具体做法。

Discovery

命令发现

The full command surface, with per-command usage hints, is one command away:

bash

langwatch commands -o json     # machine-readable catalog of every command
langwatch help-tree            # compact annotated tree (fits in context)
langwatch <group> --help       # flags for one group

Pass

--agent

to any command for compact single-line JSON with colour and spinners off (the CLI sets it by itself under Claude Code, Cursor, Copilot CLI and Amazon Q).

只需一个命令即可获取完整命令列表及各命令的使用提示：

bash

langwatch commands -o json     # machine-readable catalog of every command
langwatch help-tree            # compact annotated tree (fits in context)
langwatch <group> --help       # flags for one group

为任意命令传入

--agent

，即可获得紧凑的单行 JSON，并关闭颜色和加载指示器（在 Claude Code、Cursor、Copilot CLI 和 Amazon Q 环境下，CLI 会自动设置该参数）。