langsmith-trace-analyzer

Compare original and translation side by side

🇺🇸

Original

English

🇨🇳

Translation

Chinese

LangSmith Trace Analyzer

LangSmith 追踪数据分析工具

Use this skill to move from raw LangSmith traces to actionable debugging/evaluation insights.

使用该技能可将原始LangSmith追踪数据转化为可用于调试和评估的实用见解。

Quick Start

快速工作流

bash

undefined

bash

undefined

Install dependencies

安装依赖

uv pip install langsmith langsmith-fetch

Auth

身份验证

export LANGSMITH_API_KEY=<your_langsmith_api_key>

undefined

export LANGSMITH_API_KEY=<your_langsmith_api_key>

undefined

Fast workflow

快速流程

Download traces with

scripts/download_traces.py

(or

scripts/download_traces.ts

Analyze downloaded JSON with
```
scripts/analyze_traces.py
```
.
Load targeted references only when needed:
- ```
references/filtering-querying.md
```
  for query/filter syntax
- ```
references/analysis-patterns.md
```
  for deeper diagnostics
- ```
references/benchmark-analysis.md
```
  for benchmark-specific workflows

使用

scripts/download_traces.py

（或

scripts/download_traces.ts

）下载追踪数据。

使用
```
scripts/analyze_traces.py
```
分析下载的JSON数据。
仅在需要时加载针对性参考文档：
- ```
references/filtering-querying.md
```
  ：查询/过滤语法
- ```
references/analysis-patterns.md
```
  ：深度诊断方法
- ```
references/benchmark-analysis.md
```
  ：基准测试专属工作流

Decision Guide

决策指南

Known trace IDs
Use
```
langsmith-fetch trace <id>
```
directly, or
```
--trace-ids
```
in downloader scripts.
Need to discover traces first
Use LangSmith SDK
```
list_runs/listRuns
```
with filters, then download selected trace IDs.
Need aggregate insights
Run
```
analyze_traces.py
```
for summary stats, patterns, and passed-vs-failed comparisons.

已知追踪ID
直接使用
```
langsmith-fetch trace <id>
```
，或在下载脚本中使用
```
--trace-ids
```
参数。
需先发现追踪记录
使用LangSmith SDK的
```
list_runs/listRuns
```
方法搭配过滤器，然后下载选定的追踪ID。
需要聚合见解
运行
```
analyze_traces.py
```
获取汇总统计数据、模式分析以及通过与失败案例的对比结果。

Core Workflows

核心工作流

1) Download and organize traces

1) 下载并整理追踪数据

Python:

bash

uv run skills/langsmith-trace-analyzer/scripts/download_traces.py \
  --project "my-project" \
  --filter "job_id=abc123" \
  --last-hours 24 \
  --limit 100 \
  --output ./traces \
  --organize

TypeScript:

bash

ts-node skills/langsmith-trace-analyzer/scripts/download_traces.ts \
  --project "my-project" \
  --filter "job_id=abc123" \
  --last-hours 24 \
  --limit 100 \
  --output ./traces

Output layout:

text

traces/
├── manifest.json
└── by-outcome/
    ├── passed/
    ├── failed/
    └── error/
        ├── GraphRecursionError/
        ├── TimeoutError/
        └── DaytonaError/

Notes:

Python script supports
```
--organize/--no-organize
```
.
Both scripts use SDK filtering plus
```
langsmith-fetch
```
for full trace payload export.

Python：

bash

uv run skills/langsmith-trace-analyzer/scripts/download_traces.py \
  --project "my-project" \
  --filter "job_id=abc123" \
  --last-hours 24 \
  --limit 100 \
  --output ./traces \
  --organize

TypeScript：

bash

ts-node skills/langsmith-trace-analyzer/scripts/download_traces.ts \
  --project "my-project" \
  --filter "job_id=abc123" \
  --last-hours 24 \
  --limit 100 \
  --output ./traces

输出结构：

text

traces/
├── manifest.json
└── by-outcome/
    ├── passed/
    ├── failed/
    └── error/
        ├── GraphRecursionError/
        ├── TimeoutError/
        └── DaytonaError/

注意事项：

Python脚本支持
```
--organize/--no-organize
```
参数。
两个脚本均使用SDK过滤功能，结合
```
langsmith-fetch
```
导出完整追踪数据。

2) Analyze downloaded traces

2) 分析已下载的追踪数据

bash

undefined

bash

undefined

Markdown report

生成Markdown报告

uv run skills/langsmith-trace-analyzer/scripts/analyze_traces.py ./traces --output report.md

JSON output

生成JSON输出

uv run skills/langsmith-trace-analyzer/scripts/analyze_traces.py ./traces --json

Compare passed vs failed (expects by-outcome folders)

对比通过与失败案例（需存在by-outcome文件夹）

uv run skills/langsmith-trace-analyzer/scripts/analyze_traces.py ./traces --compare --output comparison.md


The analyzer reports:
- message/tool-call/token/duration summaries
- top tool usage
- anomaly patterns (high message count, repeated tools, quick failures)
- passed-vs-failed metric deltas when comparison is enabled

uv run skills/langsmith-trace-analyzer/scripts/analyze_traces.py ./traces --compare --output comparison.md


分析工具会生成以下报告内容：
- 消息/工具调用/令牌/时长汇总
- 高频工具使用情况
- 异常模式（消息数量过多、重复调用工具、快速失败等）
- 启用对比功能时，通过与失败案例的指标差异

3) Query traces correctly (SDK)

3) 正确查询追踪数据（SDK）

Use official LangSmith run filter syntax via

filter

and/or

start_time

python

from datetime import datetime, timedelta, timezone
from langsmith import Client

client = Client()

start = datetime.now(timezone.utc) - timedelta(hours=24)
filter_query = 'and(eq(metadata_key, "job_id"), eq(metadata_value, "abc123"))'

runs = client.list_runs(
    project_name="my-project",
    is_root=True,
    start_time=start,
    filter=filter_query,
)

For TypeScript:

import { Client } from "langsmith";

const client = new Client();
for await (const run of client.listRuns({
  projectName: "my-project",
  isRoot: true,
  filter: 'and(eq(metadata_key, "job_id"), eq(metadata_value, "abc123"))',
})) {
  console.log(run.id, run.status);
}

通过

filter

和/或

start_time

参数使用官方LangSmith运行实例过滤语法：

python

from datetime import datetime, timedelta, timezone
from langsmith import Client

client = Client()

start = datetime.now(timezone.utc) - timedelta(hours=24)
filter_query = 'and(eq(metadata_key, "job_id"), eq(metadata_value, "abc123"))'

runs = client.list_runs(
    project_name="my-project",
    is_root=True,
    start_time=start,
    filter=filter_query,
)

TypeScript版本：

import { Client } from "langsmith";

const client = new Client();
for await (const run of client.listRuns({
  projectName: "my-project",
  isRoot: true,
  filter: 'and(eq(metadata_key, "job_id"), eq(metadata_value, "abc123"))',
})) {
  console.log(run.id, run.status);
}

Accuracy and Schema Notes

准确性与架构说明

LangSmith run fields are commonly top-level (
```
status
```
,
```
error
```
,
```
total_tokens
```
,
```
start_time
```
,
```
end_time
```
).
Some exported traces also include nested metadata (
```
metadata
```
or
```
extra.metadata
```
) and/or
```
messages
```
.
```
analyze_traces.py
```
is resilient to multiple payload shapes, including raw array payloads.
For full conversation content, prefer downloaded trace payloads over bare
```
list_runs
```
results.

LangSmith运行实例的字段通常位于顶层（如
```
status
```
、
```
error
```
、
```
total_tokens
```
、
```
start_time
```
、
```
end_time
```
）。
部分导出的追踪数据还包含嵌套元数据（
```
metadata
```
或
```
extra.metadata
```
）和/或
```
messages
```
字段。
```
analyze_traces.py
```
可适配多种数据结构，包括原始数组格式。
如需完整对话内容，建议使用下载的追踪数据JSON，而非单纯的
```
list_runs
```
结果。

Troubleshooting

故障排除

Issue	Likely Cause	Action
`LANGSMITH_API_KEY` missing	Auth not configured	`export LANGSMITH_API_KEY=<your_langsmith_api_key>`
No runs returned	Wrong project/filter/time range	Verify project name and filter syntax
Empty/partial message arrays	Run schema differs or incomplete data	Use downloaded trace JSON and inspect `status/error` fields
JSON parse error on downloaded files	Bad/incomplete export	Re-download trace; use `--format raw` paths in scripts
Re-downloading same traces repeatedly	Existing files in nested folders	Use current scripts (they check existing files across output tree)

问题	可能原因	解决方法
缺少 `LANGSMITH_API_KEY`	未配置身份验证	执行 `export LANGSMITH_API_KEY=<your_langsmith_api_key>`
未返回任何运行实例	项目/过滤器/时间范围错误	验证项目名称和过滤器语法
消息数组为空/不完整	运行实例架构不同或数据不完整	使用下载的追踪数据JSON并检查 `status/error` 字段
下载的文件出现JSON解析错误	导出数据损坏/不完整	重新下载追踪数据；在脚本中使用 `--format raw` 路径
重复下载相同的追踪数据	嵌套文件夹中存在现有文件	使用当前版本的脚本（它们会检查输出目录中的现有文件）

Safety for Open Source

开源安全注意事项

Do not commit downloaded trace artifacts (
```
manifest.json
```
, trace JSON dumps) unless sanitized.
Trace payloads can contain user prompts, outputs, metadata, and other sensitive runtime data.
Keep this skill repository focused on scripts/templates, not production trace exports.

除非经过脱敏处理，否则请勿提交已下载的追踪数据文件（如
```
manifest.json
```
、追踪数据JSON转储文件）。
追踪数据可能包含用户提示、输出、元数据以及其他敏感运行时数据。
请确保该技能仓库仅聚焦于脚本/模板，而非生产环境追踪数据导出。

langsmith-trace-analyzer

Original

Translation

LangSmith Trace Analyzer

LangSmith 追踪数据分析工具

Quick Start

快速工作流

Install dependencies

安装依赖

Auth

身份验证

Fast workflow

快速流程

Decision Guide

决策指南

Core Workflows

核心工作流

1) Download and organize traces

1) 下载并整理追踪数据

2) Analyze downloaded traces

2) 分析已下载的追踪数据

Markdown report

生成Markdown报告

JSON output

生成JSON输出

Compare passed vs failed (expects by-outcome folders)

对比通过与失败案例（需存在by-outcome文件夹）

3) Query traces correctly (SDK)

3) 正确查询追踪数据（SDK）

Accuracy and Schema Notes

准确性与架构说明

Troubleshooting

故障排除

Safety for Open Source

开源安全注意事项

Resources

资源

scripts/

scripts/目录

references/

references/目录