agent-performance

Compare original and translation side by side

🇺🇸

Original

English

🇨🇳

Translation

Chinese

Diagnose Your Agent's Production Behavior

诊断你的Agent生产环境行为

This skill is a production diagnostician. It reads the real traffic, not the code, and answers: what is my agent actually doing out there, where is it failing, who is it annoying, and where is the money going. It is read-only on the platform: the only thing it writes is a report file.

该技能是一款生产环境诊断工具。它读取真实流量而非代码，解答以下问题：我的Agent在实际环境中到底在做什么？它在哪里出现故障？哪些用户对它不满意？成本消耗在何处？它在平台上是只读的：唯一会生成的内容是一份报告文件。

Step 1: Set up the LangWatch CLI

步骤1：设置LangWatch CLI

Use

langwatch docs <path>

to read documentation as Markdown. Some useful entry points:

bash

langwatch docs                                    # Docs index
langwatch docs integration/python/guide           # Python integration
langwatch docs integration/typescript/guide       # TypeScript integration
langwatch docs prompt-management/cli              # Prompts CLI
langwatch scenario-docs                           # Scenario docs index

Discover commands with

langwatch --help

and

langwatch <subcommand> --help

. List and get commands accept

--format json

for machine-readable output. Read the docs first instead of guessing SDK APIs or CLI flags.

If no shell is available, fetch the same Markdown over plain HTTP. Append

.md

to any docs path (e.g. https://langwatch.ai/docs/integration/python/guide.md). Index: https://langwatch.ai/docs/llms.txt. Scenario index: https://langwatch.ai/scenario/llms.txt

If anything fails or confuses you while following this skill (broken commands, docs that do not match reality, errors you had to work around), ask the user for permission and run

npx langwatch report --user-approved

with a

--title

and

--summary

(or

--session <transcript.jsonl>

) to send it to the LangWatch team. No login needed, secrets and personal data are redacted locally, and it directly shapes what gets fixed.

npx langwatch report --help

explains the options.

Projects and API keys: target a real project, not a personal one.

LangWatch has two kinds of project:

Team / shared projects: real projects inside an organization. Evaluations, experiments, prompts, datasets, simulations and instrumentation must always target one of these.
Personal projects: a private "My Workspace" scratch space tied to a single user. Never send a user's evaluations, experiments or production traces here: it is for personal exploration only and is easily confused with a real project.

And two ways to authenticate:

A project API key in
.env
(
```
LANGWATCH_API_KEY
```
): the credential everything in these skills uses. It is scoped to one real project. This is the default; prefer it unless the user explicitly asks for something else.
langwatch login --device
(AI-tools / SSO): a personal device session for wrapping coding assistants (
```
langwatch claude
```
,
```
langwatch codex
```
, …). It is NOT for evaluations, prompts, datasets, scenarios or SDK instrumentation, and it points at a personal workspace. Do not run it to set up the work in these skills.

So for anything in these skills: make sure

LANGWATCH_API_KEY

for a real, shared project is in the project's

.env

— most environments already have this provisioned. Do NOT run

langwatch login

to pick a project, and never default to a personal project. If

LANGWATCH_ENDPOINT

is set, they are self-hosted, use that endpoint instead of app.langwatch.ai.

使用

langwatch docs <path>

以Markdown格式读取文档。一些实用的入口点：

bash

langwatch docs                                    # 文档索引
langwatch docs integration/python/guide           # Python集成指南
langwatch docs integration/typescript/guide       # TypeScript集成指南
langwatch docs prompt-management/cli              # 提示词管理CLI
langwatch scenario-docs                           # 场景文档索引

使用

langwatch --help

和

langwatch <subcommand> --help

查看可用命令。列表和获取类命令支持

--format json

参数以输出机器可读格式。在尝试SDK API或CLI参数前，请先阅读文档。

如果没有Shell环境，可以通过普通HTTP获取相同的Markdown文档。在任意文档路径后添加

.md

即可（例如：https://langwatch.ai/docs/integration/python/guide.md）。文档索引：https://langwatch.ai/docs/llms.txt。场景文档索引：https://langwatch.ai/scenario/llms.txt

如果在使用该技能过程中遇到任何问题或困惑（命令失效、文档与实际不符、需要自行解决的错误），请先征得用户许可，然后运行

npx langwatch report --user-approved

并带上

--title

和

--summary

参数（或

--session <transcript.jsonl>

）将问题反馈给LangWatch团队。无需登录，敏感信息和个人数据会在本地自动脱敏，反馈内容将直接用于问题修复。

npx langwatch report --help

会解释具体选项。

项目与API密钥：请选择真实的团队项目，而非个人项目。

LangWatch有两种类型的项目：

团队/共享项目：组织内部的真实项目。评估、实验、提示词、数据集、模拟和工具集成必须始终指向这类项目。
个人项目：绑定单个用户的私人「我的工作区」临时空间。切勿将用户的评估、实验或生产追踪记录发送到这里：它仅用于个人探索，容易与真实项目混淆。

还有两种认证方式：

.env文件中的项目API密钥（
```
LANGWATCH_API_KEY
```
）：所有这类技能使用的凭证。它仅对一个真实项目生效。这是默认方式；除非用户明确要求，否则优先使用此方式。
langwatch login --device
（AI工具/单点登录）：用于包装代码助手（
```
langwatch claude
```
、
```
langwatch codex
```
等）的个人设备会话。它不能用于评估、提示词、数据集、场景或SDK集成，且指向个人工作区。请勿运行此命令来设置本技能所需的工作环境。

因此，对于本技能中的所有操作：确保真实共享项目的

LANGWATCH_API_KEY

已配置在项目的

.env

文件中——大多数环境已预先配置好此内容。不要运行

langwatch login

来选择项目，也绝不要默认使用个人项目。如果设置了

LANGWATCH_ENDPOINT

，说明是自部署环境，请使用该端点而非app.langwatch.ai。

Step 2: Baseline the Vital Signs

步骤2：评估关键指标基线

Establish the macro picture first, always comparing against the previous period (the analytics API returns both periods for every query):

bash

langwatch status                                                  # Resource counts and project overview
langwatch analytics query --metric trace-count --format json      # Volume trend, last 7 days
langwatch analytics query --metric total-cost --format json       # Spend trend
langwatch analytics query --metric avg-latency --format json      # Latency trend
langwatch analytics query --metric p95-latency --format json      # Tail latency
langwatch analytics query --metric total-tokens --format json     # Token consumption
langwatch analytics query --metric eval-pass-rate --format json   # Quality trend, if evaluators exist

Then slice the same metrics to find WHERE the numbers come from:

bash

langwatch analytics query --metric total-cost --group-by metadata.model --format json
langwatch analytics query --metric trace-count --group-by metadata.labels --format json
langwatch analytics query --metric p95-latency --group-by metadata.model --format json

Widen with

--start-date

(ISO) to 30 days when trends look suspicious: a gradual drift only shows on longer windows. Run

langwatch analytics query --help

for every preset and flag.

An empty metric is a coverage note, never the end of the road. An empty

eval-pass-rate

means there were no evaluator runs in the selected window; it says nothing about the traffic itself, which

trace-count

total-cost

, and the latency metrics still describe. For an open "what has my agent been up to?", answer from whichever sources HAVE data, production traces first, then simulation runs: share a few concrete observations (volume, the kinds of requests coming in, errors, cost or latency movements, one or two example traces), then end with one short line inviting the user to name what to dig into more deeply ("Say which of these to dig into and I'll go deeper."). Never end the conversation on "no evaluation data" alone when the project has traces.

首先建立宏观视图，始终与上一周期进行对比（分析API会为每个查询返回两个周期的数据）：

bash

langwatch status                                                  # 资源统计与项目概览
langwatch analytics query --metric trace-count --format json      # 近7天流量趋势
langwatch analytics query --metric total-cost --format json       # 成本消耗趋势
langwatch analytics query --metric avg-latency --format json      # 平均延迟趋势
langwatch analytics query --metric p95-latency --format json      # P95尾部延迟
langwatch analytics query --metric total-tokens --format json     # Token消耗情况
langwatch analytics query --metric eval-pass-rate --format json   # 质量趋势（若存在评估器）

然后对相同指标进行细分，找出数据来源：

bash

langwatch analytics query --metric total-cost --group-by metadata.model --format json
langwatch analytics query --metric trace-count --group-by metadata.labels --format json
langwatch analytics query --metric p95-latency --group-by metadata.model --format json

当趋势看起来异常时，使用

--start-date

（ISO格式）将时间范围扩大到30天：渐进式的变化只有在更长的时间窗口中才会显现。运行

langwatch analytics query --help

查看所有预设参数和选项。

指标为空仅说明数据覆盖情况，而非终点。

eval-pass-rate

为空意味着所选时间窗口内没有评估器运行；这并不影响流量本身的描述，

trace-count

、

total-cost

和延迟指标仍能反映真实情况。如果用户询问「我的Agent最近在做什么？」，请从有数据的来源（优先生产追踪记录，其次是模拟运行）回答：分享一些具体观察（流量规模、请求类型、错误情况、成本或延迟变化、1-2个示例追踪记录），最后用简短一句话邀请用户指定进一步挖掘的方向（「请告诉我你想深入了解哪方面，我会进行更详细的分析。」）。当项目有追踪记录时，绝不要只以「无评估数据」结束对话。

Step 3: Export the Evidence and Mine It

步骤3：导出证据并分析

Aggregates say WHAT changed; only the traces say WHY. Export a large sample and analyze it locally:

bash

langwatch trace export --format jsonl --limit 1000 --origin application -o traces.jsonl
langwatch trace export --format jsonl --limit 1000 --origin application --start-date <30d-ago> --end-date <14d-ago> -o traces-before.jsonl

--origin application

scopes the sample to real production traffic (it includes traces with no recorded origin). Evaluation, simulation, playground, gateway, and langy traces would pollute the picture of what the agent does for users; include those origins (comma-separated) only when they are the subject of the question.

Write small local scripts (python3 or jq) over the JSONL to compute, at minimum:

Failure patterns: cluster error traces by error message and by input shape. Which user intents fail most?
Dissatisfied users: traces with negative feedback or angry language in inputs ("this is wrong", "that's not what I asked", repeated rephrasing of the same question in a thread). Check annotations on candidate traces too: thumbs down and reviewer comments are gold.
Token and cost hotspots: distribution of tokens per trace; the p99 tail; which metadata slice (model, label, user) concentrates the spend; prompts that balloon context.
Edge cases: inputs far from the common distribution (very long, empty, non-primary language, unusual formats) and how the agent handled them.
Behavior changes: compare the recent window against the older export: output length, tool usage mix, model mix, refusal rate, latency. Anything that moved, find the first day it moved.
Outliers: the single weirdest traces by duration, cost, span count, and output size. Read them individually.

bash

langwatch trace search -q "<keyword from a pattern>" --origin application --limit 10 --format json   # Chase a specific pattern
langwatch trace get <traceId>                                                   # Read a representative trace in full
langwatch trace get <traceId> -f json                                           # Every span, token count, and timing

For every pattern you claim, keep 2-3 example trace IDs as evidence. Never report a pattern without example traces behind it.

聚合数据能说明什么发生了变化；只有追踪记录能解释为什么变化。导出大量样本并在本地分析：

bash

langwatch trace export --format jsonl --limit 1000 --origin application -o traces.jsonl
langwatch trace export --format jsonl --limit 1000 --origin application --start-date <30d-ago> --end-date <14d-ago> -o traces-before.jsonl

--origin application

参数将样本范围限定为真实生产流量（包含未记录来源的追踪记录）。评估、模拟、沙箱、网关和langy类型的追踪记录会干扰Agent面向用户的真实表现分析；只有当这些来源是问题的核心时，才需要包含它们（用逗号分隔）。

编写小型本地脚本（python3或jq）处理JSONL文件，至少计算以下内容：

故障模式：按错误信息和输入类型聚类错误追踪记录。哪些用户意图最容易失败？
不满意的用户：包含负面反馈或输入中有愤怒语言的追踪记录（例如「这是错的」「这不是我要的」、同一问题重复表述）。同时检查候选追踪记录的注释：差评和审核者评论是宝贵的信息。
Token与成本热点：每个追踪记录的Token分布；P99尾部情况；哪些元数据维度（模型、标签、用户）集中了成本消耗；导致上下文膨胀的提示词。
边缘案例：与常见分布差异较大的输入（极长、空内容、非主要语言、特殊格式）以及Agent的处理方式。
行为变化：对比近期窗口与早期导出的数据：输出长度、工具使用组合、模型组合、拒绝率、延迟。找出任何发生变化的指标及其首次变化的日期。
异常情况：在时长、成本、跨度数量和输出大小方面最异常的单个追踪记录。逐一查看这些记录。

bash

langwatch trace search -q "<pattern中的关键词>" --origin application --limit 10 --format json   # 追踪特定模式
langwatch trace get <traceId>                                                   # 完整查看代表性追踪记录
langwatch trace get <traceId> -f json                                           # 获取所有跨度、Token计数和时间数据

对于你提出的每个模式，保留2-3个示例追踪记录ID作为证据。绝不要在没有示例追踪记录支持的情况下报告模式。

Step 4: Build the Report

步骤4：生成报告

Write a single self-contained

agent-performance-report.html

in the project root (inline CSS, no external assets) with:

Executive summary: the 3-5 findings that matter, each one sentence with its magnitude ("34% of errors come from date parsing on non-English inputs")
One section per finding: the metric evidence (small tables, before/after numbers), what it means, and links to example traces so every claim is verifiable in one click
A cost breakdown section, a reliability section, and a user-satisfaction section, even when healthy: say what was checked and that it looks fine
A closing "recommended next steps" section ranked by impact

Trace links:

langwatch trace get

returns the platform URL for each trace; use those URLs directly. Anyone on the project team can open them.

Open the report path for the user and also summarize the top findings directly in the conversation, leading with the numbers.

在项目根目录编写一个独立的

agent-performance-report.html

文件（使用内联CSS，无外部资源），包含以下内容：

执行摘要：3-5个最重要的发现，每个发现用一句话说明并标注量级（例如「34%的错误来自非英语输入的日期解析」）
每个发现对应一个章节：包含指标证据（小型表格、前后对比数据）、含义解释，以及示例追踪记录链接，让每个结论都能一键验证
成本细分章节、可靠性章节和用户满意度章节，即使数据表现良好也要包含：说明已检查的内容并确认状态正常
结尾的「建议下一步行动」章节，按影响程度排序

追踪记录链接：

langwatch trace get

会返回每个追踪记录的平台URL；直接使用这些URL即可。项目团队中的任何人都可以打开这些链接。

为用户打开报告路径，并在对话中直接总结最重要的发现，优先展示数据。

Step 5: Hand Off to Improvement

步骤5：移交至优化环节

Diagnosis without treatment is just bad news. If the

agent-improve

skill is installed, run it on the findings right away: it turns each finding into tested hypotheses, scenario tests, evaluators, and PR-ready changes. Pass along the report — agent-improve uses these findings and trace examples as its evidence base.

只诊断不给出解决方案就只是坏消息。如果已安装

agent-improve

技能，请立即基于发现结果运行它：它会将每个发现转化为经过测试的假设、场景测试、评估器和可直接提交PR的变更。将报告一并传递——agent-improve会将这些发现和示例追踪记录作为证据基础。

Common Mistakes

常见错误

Do NOT modify the agent's code, prompts, or any platform resource; this skill is read-only plus one report file
Do NOT report a pattern without linked example traces; unverifiable claims are worthless
Do NOT rely on aggregates alone; always read at least a handful of full traces per finding, the surprise is always in the details
Do NOT analyze only the happy window; without a before/after comparison you cannot see behavior change
Do NOT dump raw JSON at the user; the deliverable is the diagnosis and the report, written in plain language with numbers
Do NOT stop at an empty evaluation metric; when evaluations have no data, the answer comes from the traces (and simulation runs), with a closing invitation to dig deeper
Do NOT mix origins blindly; questions about production behavior are answered from
```
--origin application
```
traffic
If the CLI returns an error, report the user-facing consequence (what couldn't be determined and why in plain terms), not the raw error text — an activity card already shows the underlying failure

请勿修改Agent的代码、提示词或任何平台资源；本技能是只读的，仅会生成一份报告文件
请勿在没有链接示例追踪记录的情况下报告模式；无法验证的结论毫无价值
请勿仅依赖聚合数据；对于每个发现，至少查看少量完整的追踪记录，细节中往往藏着意外
请勿仅分析表现良好的时间窗口；没有前后对比就无法发现行为变化
请勿向用户输出原始JSON；交付物应该是用通俗易懂的语言和数据呈现的诊断结果与报告
请勿在评估指标为空时停止分析；当评估无数据时，答案来自追踪记录（和模拟运行），并以邀请进一步挖掘的话语结束对话
请勿盲目混合不同来源的追踪记录；关于生产环境行为的问题应使用
```
--origin application
```
流量来解答
如果CLI返回错误，请报告对用户的影响（用通俗易懂的语言说明无法确定的内容及原因），而非原始错误文本——活动卡片已显示底层故障信息