agent-observability-replay-trace
Compare original and translation side by side
🇺🇸
Original
English🇨🇳
Translation
ChineseReplay a trace against local code
针对本地代码重放Trace
A fast iteration loop on a single production trace: take a trace whose output a developer didn't like,
optionally change the code, re-run it against their LOCAL code, and show a concise diff of old vs new
output — repeating until they're happy. Assumes nothing about the project's layout.
Invoked from the developer's coding agent: .
With no modification, do the replay + diff only (a reproduce/regression check), then offer to enter the loop.
/agent-observability-replay-trace <trace-id> [<changes to test>]This file is the workflow spine — terse on purpose. The depth lives in (trace
backend + pup flags, the runner contract, export mode, polling, the trace-link scoping fix) and
(making a deployed-only app locally runnable). Read before you touch
pup or generate the runner.
references/details.mdreferences/local-setup.mddetails.mdWriting code — keep comments minimal to none. Everything you generate or edit (the annotation, the
runner's entries, a local harness, iteration edits) should match the surrounding code and
carry no unnecessary comments — don't narrate what the code plainly does; add a comment only for a
genuinely non-obvious why.
ENTRYPOINTSIntent tagging: On every MCP tool call, prefix with (a short per-run id, generated once and reused for every call) followed by a description of why the tool is being called. On the first MCP tool call only, use instead (note the suffix). Example first call: . pup-CLI calls carry no , so this applies only on the MCP path.
datadog-llmotelemetry.intentskill:agent-observability-replay-trace[<inv_id>] — skill:agent-observability-replay-trace:start[<inv_id>] — :startskill:agent-observability-replay-trace:start[3a9f1c2b] — fetch the original trace's baseline outputtelemetry.intent这是针对单个生产环境trace的快速迭代循环:获取开发者不满意其输出的trace,
可选择性修改代码,在本地代码上重新运行,并展示新旧输出的简洁差异——重复此过程直至开发者满意。不依赖项目的布局结构。
从开发者的编码代理调用:。
若未做任何修改,则仅执行重放+对比操作(用于复现/回归检查),随后提供进入循环的选项。
/agent-observability-replay-trace <trace-id> [<要测试的更改>]本文件为工作流核心——刻意保持简洁。详细内容请见 (trace后端+pup参数、运行器契约、导出模式、轮询、trace链接范围修复)和 (使仅部署的应用可在本地运行)。在使用pup或生成运行器前,请先阅读 。
references/details.mdreferences/local-setup.mddetails.md编写代码——尽量减少注释甚至不写注释。 你生成或编辑的所有内容(注解、运行器的条目、本地 harness、迭代编辑)应与周围代码风格一致,且不包含不必要的注释——无需解释代码明面上的功能;仅当存在真正非显而易见的「原因」时才添加注释。
ENTRYPOINTS意图标记: 在每次 MCP工具调用时,在前添加前缀 (每个运行的短ID,生成一次后重复用于所有调用),后跟调用该工具的原因描述。仅在第一次MCP工具调用时,改用前缀 (注意后缀)。示例首次调用:。pup-CLI调用无需添加,因此仅适用于MCP路径。
datadog-llmotelemetry.intentskill:agent-observability-replay-trace[<inv_id>] — skill:agent-observability-replay-trace:start[<inv_id>] — :startskill:agent-observability-replay-trace:start[3a9f1c2b] — fetch the original trace's baseline outputtelemetry.intentInteraction model — selector gates, never a hard stop
交互模型——选择器网关,永不强制终止
This is a live loop. At every decision point present the choices as an selector (the
plan-mode-style menu), not a plain question that ends your turn. Two gates: (a) after you propose code
changes, before replaying; (b) after each diff. The selector's free-text option lets the user type detail
(what to refine) inline — act on it directly, don't ask a follow-up. Keep re-presenting after every replay
until they pick "stop here".
AskUserQuestion这是一个实时循环。在每个决策点,将选项以**选择器**(计划模式风格菜单)呈现,而非会终止流程的普通问题。两个网关:(a) 在你提出代码更改后、重放前;(b) 在每次对比后。选择器的自由文本选项允许用户输入细节(要优化的内容)——直接执行操作,无需后续追问。在每次重放后重新呈现选择器,直至用户选择「在此停止」。
AskUserQuestionScope — check first
范围——先检查
- Traced with / LLM Obs (an
ddtrace+ a discoverable entrypoint). Python is first-class; other languages work but you write the runner to the contract in their SDK/build tooling.ml_app - JSON-serializable entrypoint input, and a callable seam for the root span (see step 3.5 — not a binary "is it runnable?"; deployed-only apps often still expose a plain callable).
- A trace-access backend — the MCP (used when present) or the
datadog-llmoCLI (fallback, and the easier install if you have neither) (step 0).pup - Credentials: +
DD_API_KEY+ provider key(s). NotDD_SITE— plain trace, not an Experiment (that'sDD_APP_KEY).agent-observability-replay-experiment - Side effects, irreversible: replaying re-runs real code (model spend + real writes), and LLM Obs
traces cannot be deleted — a mis-scoped replay (wrong ml_app) permanently pollutes the production app's
dashboards/eval sets. That's why the isolation (steps 4/6/7) is load-bearing, not tidy. Warn before the first replay.
<ml_app>-local
- 使用/ LLM Obs追踪(包含
ddtrace+ 可发现的入口点)。Python为一等支持; 其他语言也可使用,但需根据其SDK/构建工具编写符合契约的运行器。ml_app - 入口点输入可JSON序列化,且根span存在可调用的接缝(见步骤3.5——并非简单的「是否可运行」;仅部署的应用通常仍会暴露普通的可调用接口)。
- 具备trace访问后端——MCP(若存在则使用,默认选项,读取功能更丰富:结构化树+
datadog-llmo);否则使用**content_infoCLI**(备选方案,若两者都没有则更容易安装)(步骤0)。pup - 凭据:+
DD_API_KEY+ 提供商密钥。请勿使用DD_SITE——仅针对普通trace,而非实验(实验对应DD_APP_KEY)。agent-observability-replay-experiment - 副作用、不可逆:重放会重新运行真实代码(模型成本+真实写入),且LLM Obs trace无法删除——范围错误的重放(错误的ml_app)会永久污染生产应用的仪表盘/评估集。这就是隔离(步骤4/6/7)至关重要的原因,而非仅仅是整洁性要求。在首次重放前发出警告。
<ml_app>-local
Workflow
工作流
0. Ensure a trace-access backend
0. 确保具备trace访问后端
Pick, in order: (1) the MCP if tools are present — the default (slightly
richer for reads: structured tree + ); (2) else if installed and targets
the app's org; (3) else the user has neither → guide the pup install (it's easier to set up than the
MCP, so recommend pup here):
mcp__datadog-llmo-mcp__*content_infopuppup authbrew tap datadog-labs/pack && brew install datadog-labs/pack/pup
pup auth login(MCP alternative: ; see https://docs.datadoghq.com/bits_ai/mcp_server/setup/.) Don't proceed without a backend.
The backend↔operation mapping and pup's exact flags/gotchas are in — read that section before
using pup. Two pup musts: (1) results come back at or top-level
(varies by version/) — parse whichever is present, or you get zero hits on an ingested
trace (a silent false negative, step 7); (2) check token expiry (), not just that auth
exists — expiry mid-loop looks like "trace not found."
claude mcp add --scope user --transport http "datadog-llmo-mcp" "https://mcp.datadoghq.com/api/unstable/mcp-server/mcp?toolsets=llmobs"details.mddata.spans[]spans[]--no-agentpup auth status按以下顺序选择:(1) 若存在工具,则使用MCP——默认选项(读取功能更丰富:结构化树+);(2) 否则若已安装且指向应用的组织,则使用**;(3) 否则用户两者都没有→引导安装pup**(设置比MCP更简单,因此在此推荐pup):
mcp__datadog-llmo-mcp__*content_infopuppup authpupbrew tap datadog-labs/pack && brew install datadog-labs/pack/pup
pup auth login(MCP备选方案:;详见https://docs.datadoghq.com/bits_ai/mcp_server/setup/。)无后端则无法继续。
后端与操作的映射以及pup的具体参数/注意事项见——在使用pup前请阅读该部分。pup的两个必须事项:(1) 结果返回在**或顶级(因版本/而异)——解析存在的任意一个**,否则会在已摄入的trace上得到零结果(静默假阴性,步骤7);(2) 检查令牌过期(),而不仅仅是检查是否存在授权——循环中途过期会显示「trace未找到」。
claude mcp add --scope user --transport http "datadog-llmo-mcp" "https://mcp.datadoghq.com/api/unstable/mcp-server/mcp?toolsets=llmobs"details.mddata.spans[]spans[]--no-agentpup auth status1. Parse the command
1. 解析命令
<trace-id>ml_appLLMObs.enable(ml_app=…)DD_LLMOBS_ML_APP<trace-id>LLMObs.enable(ml_app=…)DD_LLMOBS_ML_APPml_app2. Fetch the trace + locate the baseline
2. 获取trace + 定位基准
Fetch via the backend; note (drives step 7), the , and
/ if present. Locate the baseline field — it's not always the
root output: the value the developer dislikes may be a tool-call input or an intermediate output several
levels deep, and the app may post-process it before the span records it. Pick the field the code change can
actually move, or the delta drowns in noise.
total_duration_mstrace_urlmetadata.replay_inputreplay_entrypoint通过后端获取trace;记录(用于步骤7)、,以及若存在的/。定位基准字段——它并非总是根输出:开发者不满意的值可能是工具调用输入或深层的中间输出,应用可能在span记录前对其进行后处理。选择代码更改实际能影响的字段,否则差异会被噪音淹没。
total_duration_mstrace_urlmetadata.replay_inputreplay_entrypoint2.5. Check for fan-out
2.5. 检查是否存在扇出
If the root span fans out into repeated sibling subtrees (a batch/map over N parallel sub-runs), the
change under test is usually visible in a single branch — replaying the whole root costs ~N× spend and
time for no extra signal. Offer to replay one representative branch; log what you skipped. Pick deliberately: the cheapest
branch that reached the terminal / side-effecting tool (most branches are no-ops that prove nothing), and
reconstruct its input from the child span's input, not the root's. Full root only if the change is
inherently cross-branch.
若根span扇出为重复的兄弟子树(对N个并行子运行进行批量/映射处理),则测试的更改通常在单个分支中可见——重放整个根会花费约N倍的成本和时间,且无额外信号。提供重放一个代表性分支的选项;记录跳过的内容。刻意选择:成本最低且到达终端/有副作用工具的分支(大多数分支是无操作,无法证明任何内容),并从子span的输入重构其输入,而非根span的输入。仅当更改本质上跨分支时才重放完整根。
3. Resolve the entrypoint + input
3. 解析入口点 + 输入
- Entrypoint: if present; else infer from the root span (name/kind) + code and confirm with the user.
metadata.replay_entrypoint - Input: if present; else derive a suggested input (prefer the code signature — the rendered prompt is lossy) and have the user confirm/edit.
metadata.replay_input
- 入口点: 若存在则使用;否则从根span(名称/类型)+ 代码推断,并与用户确认。
metadata.replay_entrypoint - 输入: 若存在则使用;否则推导建议的输入(优先选择代码签名——渲染的提示会丢失信息),并让用户确认/编辑。
metadata.replay_input
3.5. Ensure a local run path (find the innermost callable seam)
3.5. 确保本地运行路径(找到最内层的可调用接缝)
Ask "what is the innermost callable seam for this root span, and can I call it directly with JSON?" — not
"is the app runnable?". Already directly callable → skip, continue. Buried under a handler/service
(deployed-only, no local , live-infra coupling) → follow (detect →
propose → approve → build). The common middle case — a deployed service whose core logic is already a plain
callable (ports-and-adapters) — just extract/call that seam; full local-setup is overkill.
__main__references/local-setup.md询问**「此根span的最内层可调用接缝是什么,我能否直接用JSON调用它?」——而非「应用是否可运行?」。若已可直接调用→跳过,继续。若被埋在处理器/服务下(仅部署,无本地,依赖实时基础设施)→遵循**(检测→提议→批准→构建)。常见的中间情况——仅部署的服务其核心逻辑已是普通的可调用接口(端口与适配器模式)——只需提取/调用该接缝;完整的本地设置是过度设计。
__main__references/local-setup.md4. Ensure the two persistent artifacts (one-time setup)
4. 确保两个持久化工件(一次性设置)
- a) In-entrypoint annotation on the app's real entrypoint, so all future traces (production too)
self-describe. Stamp it at span start, not the success/deferred-finish path — a failed run must still
carry (those are the ones you most want to replay):
replay_inputNopythonLLMObs.annotate(span=span, metadata={"replay_entrypoint": "<stable id>", "replay_input": <extractor>})— the original trace is the baseline. (Non-Python annotate APIs differ — e.g. Goreplay_output; seespan.Annotate(llmobs.WithAnnotatedMetadata(...)).)details.md - Isolation pre-flight (before writing the runner): grep the entrypoint's call path for per-span/
per-call ml_app overrides (Go ; Python
llmobs.WithMLAppon a decorator or inml_app=). Those beat the init-levelLLMObs.annotate, so the app's spans can still land in production — tracer-level config is not proof of isolation. If any exist, the app's ml_app must resolve from env so-localwins.-local - b) The runner — satisfies the language-independent runner contract in (load env → derive
details.md→ dispatch one entrypoint on JSON → flush on every exit path incl. errors → refuse to start unless ml_app ends in<ml_app>-local→ print the-localml_app). Python: copy-localand fillscripts/replay_runner_template.py. Other languages: write to the contract — don't assume the Python API carries over (Go APIs + export-mode gotchas inENTRYPOINTS), and where the language has no in-process dotenv add a run wrapper (artifact c) that sources the project env, unsets ambient provider vars, and exports thedetails.mdoverride. Infer + confirm the run command; follow the host repo's build-file conventions (Bazel/Gazelle →-local, run Gazelle, build before replay).cmd/<name>/
- a) 入口点注解 添加到应用的真实入口点,以便所有未来的trace(包括生产环境)都能自我描述。在span开始时添加,而非成功/延迟完成路径——失败的运行仍需携带(这些是你最想重放的运行):
replay_input无需添加pythonLLMObs.annotate(span=span, metadata={"replay_entrypoint": "<stable id>", "replay_input": <extractor>})——原始trace即为基准。(非Python的注解API有所不同——例如Go的replay_output;详见span.Annotate(llmobs.WithAnnotatedMetadata(...))。)details.md - 隔离预检查(编写运行器前): 搜索入口点的调用路径,查找每个span/每次调用的ml_app覆盖(Go的;Python的装饰器上的
llmobs.WithMLApp或ml_app=中的LLMObs.annotate)。这些设置优先级高于初始化级别的ml_app=,因此应用的span仍可能进入生产环境—— tracer级别的配置不能作为隔离的证明。若存在此类设置,应用的ml_app必须从环境变量解析,以便-local生效。-local - b) 运行器——满足中的语言无关运行器契约(加载环境变量→推导
details.md→根据JSON调度一个入口点→在所有退出路径(包括错误)上刷新→除非ml_app以<ml_app>-local结尾否则拒绝启动→打印-localml_app)。Python: 复制-local并填充scripts/replay_runner_template.py。其他语言: 编写符合契约的代码——不要假设Python API可直接复用(Go API+导出模式注意事项见ENTRYPOINTS),若语言无进程内dotenv,则添加运行包装器(工件c),用于加载项目环境变量、取消设置环境中的提供商变量,并导出details.md覆盖。推断并确认运行命令;遵循宿主仓库的构建文件约定(Bazel/Gazelle→-local,运行Gazelle,重放前构建)。cmd/<name>/
5. (If a change was requested) edit, then gate
5.(若请求更改)编辑,然后进入网关
Make the code changes, show the developer the diff of your changes, then an selector:
Replay now / Adjust the changes first / Cancel. Only replay on "Replay now".
AskUserQuestion进行代码更改,向开发者展示更改的差异,然后显示选择器:立即重放 / 先调整更改 / 取消。仅在选择「立即重放」时执行重放。
AskUserQuestion6. Replay
6. 重放
Before the first replay: warn (re-running is real — model spend + real writes), and sanitize the
environment. The coding agent's own env ( / set by Claude
Code, and other provider keys) can make the app's SDK bypass its configured model gateway — a fidelity
gap invisible in the diff. Unset ambient provider vars by default and report that you did (don't
just ask); grep the app for its own ambient-key guards. Also verify the credential's org matches the
trace's org — a mismatch ships the replay somewhere you can't query (looks like ingest lag).
On confirmation, record and run — source the project's env file, never inline secrets (the marker
tag is fine on the command; inline is blocked by the permission classifier and leaks to
history/transcript — use the wrapper/env-file):
ANTHROPIC_API_KEYANTHROPIC_BASE_URLt0DD_API_KEY=<value>DD_TAGS=replay_run_id:<unique-id> <run cmd or wrapper> --entrypoint <id> --input-file <path>The runner emits under (idempotent, so replays never pollute production) and prints that
name — poll for the new trace under it.
<ml_app>-local首次重放前:发出警告(重新运行会执行真实操作——模型成本+真实写入),并清理环境。编码代理自身的环境变量( / 由Claude Code设置,以及其他提供商密钥)可能导致应用的SDK绕过其配置的模型网关——这种保真度差距在差异中不可见。默认取消设置环境中的提供商变量并报告已执行此操作(不要仅询问);搜索应用中是否存在自身的环境密钥防护。同时验证凭据的组织与trace的组织匹配——不匹配会导致重放发送到无法查询的位置(看起来像是摄入延迟)。
确认后,记录并运行——加载项目的环境文件,切勿内联密钥(标记可在命令中使用;内联会被权限分类器阻止,并泄露到历史/转录中——使用包装器/环境文件):
ANTHROPIC_API_KEYANTHROPIC_BASE_URLt0DD_API_KEY=<value>DD_TAGS=replay_run_id:<unique-id> <运行命令或包装器> --entrypoint <id> --input-file <路径>运行器在**下生成trace(幂等,因此重放永远不会污染生产环境)并打印该名称——在该名称下轮询新trace**。
<ml_app>-local7. Wait for the new trace
7. 等待新trace
- Runner subprocess timeout = .
max(120s, ~3 × total_duration_ms) - Ingest poll: after it returns, poll the backend every ~5s up to ~2 min for the tag under
replay_run_id(pup:<ml_app>-local, plain--query "replay_run_id:<id>"). Before ever reporting "not found," re-query with no tag filter (justkey:value+ window): if that returns spans, your filter/parse/scope is wrong — not ingestion. A false "no trace" reads as normal and invites a wasteful re-run.<ml_app>-local - Verify isolation on each hit — a tag match is NOT proof. /tag matching can return a span whose real
--queryis a different app (theml_appfilter gets ignored). Read--ml-appoff every returned span and assert it ends inml_appbefore reporting a clean replay — otherwise you report "clean replay under-local" while the trace is actually in production (which you can't undo). This false confidence is worse than the false negative. Don't hard-fail on timeout; offer to keep waiting.-local
- 运行器子进程超时 = 。
max(120s, ~3 × total_duration_ms) - 摄入轮询: 运行器返回后,每
5秒轮询后端一次,最多2分钟,查找下带有<ml_app>-local标签的trace(pup:replay_run_id,普通--query "replay_run_id:<id>"格式)。在报告「未找到」之前,务必不带标签过滤器重新查询(仅key:value+时间窗口):若返回span,则你的过滤器/解析/范围有误——并非摄入问题。错误的「无trace」会被视为正常情况,导致不必要的重新运行。<ml_app>-local - 每次命中时验证隔离——标签匹配并非证明。 /标签匹配可能返回实际
--query为不同应用的span(ml_app过滤器被忽略)。读取每个返回span的--ml-app并断言其以ml_app结尾,然后报告重放成功——否则你会报告「在-local下成功重放」,但实际trace在生产环境中(无法撤销)。这种错误的信心比假阴性更糟糕。超时后不要强制终止;提供继续等待的选项。-local
8. Diff (with links to both traces)
8. 对比(附带两个trace的链接)
Concise summary of how the new output differs from the old — meaningful differences only. Note live-world
drift; and because any nondeterministic agent varies run-to-run, default to two replays (diff-only mode
too, not just model-facing edits) and use replay-to-replay comparison — if the two local runs differ
from each other about as much as from production, the delta is sampling variance, not your change. If the
replay disables a side-effecting integration (dry-run), that integration's subtree is absent — exclude
it from both sides before comparing span counts, or the structural diff is junk. Lead the diff with both
trace links:
- Old: verbatim — but under fan-out (you replayed one branch) link the branch span, not the whole-root url.
trace_url - New (replay): must carry or it opens empty — and the
ml_app=<ml_app>-localis an org-switch wrapper (trace_url), so inject…/switch_to_user/<id>?next=<encoded /llm/traces …>&flow=org_switchinto the decodedml_app=<ml_app>-localquery and re-encode; do NOT append to the outer URL (mechanics innext). Browser-unverifiable from here — confirm once it opens non-empty.details.md
简洁总结新输出与旧输出的差异——仅展示有意义的差异。注意现实世界的偏差;由于任何非确定性代理每次运行都会变化,默认执行两次重放(不仅针对模型相关编辑,也针对仅对比模式)并使用重放间对比——若两次本地运行之间的差异与生产环境的差异大致相同,则差异是采样方差,而非你的更改导致。若重放禁用了有副作用的集成(试运行),则该集成的子树不存在——在比较span数量前将其从双方排除,否则结构差异毫无意义。对比开头附带两个trace链接:
- 旧: 原样保留——但在扇出情况下(你重放了一个分支)链接到分支span,而非整个根的url。
trace_url - 新(重放): 必须携带,否则会打开空页面——且
ml_app=<ml_app>-local是组织切换包装器(trace_url),因此将…/switch_to_user/<id>?next=<encoded /llm/traces …>&flow=org_switch注入解码后的ml_app=<ml_app>-local查询参数并重新编码;不要附加到外部URL(机制见next)。在此无法验证浏览器是否能打开——确认打开后非空即可。details.md
9. Gate — iterate, or stop on a broken harness
9. 网关——迭代,或在harness故障时停止
Harness-failure gate (before the diff): if a replay reveals the harness is wrong — trace landed under
the wrong ml_app, no trace after the step-7 sanity checks, missing flush, or auth/org misrouted — do NOT
proceed to a diff on bad data. Stop and present a selector to fix the harness (re-scope ml_app / add flush
/ fix env) and re-replay.
Otherwise, after the diff, an selector: Looks good — stop here (finish; leave the edits
in the working tree) / Make more changes (free-text inline → back to step 5). Re-present after every
replay; end only on "stop here".
AskUserQuestionHarness故障网关(对比前): 若重放发现harness错误——trace落在错误的ml_app下,经过步骤7的检查后仍无trace,缺少刷新,或授权/组织路由错误——不要基于错误数据进行对比。停止并呈现选择器以修复harness(重新设置ml_app范围/添加刷新/修复环境)并重新重放。
否则,对比后显示选择器:看起来不错——在此停止(结束;将编辑保留在工作树中) / 进行更多更改(自由文本输入→返回步骤5)。每次重放后重新呈现选择器;仅在选择「停止」时结束。
AskUserQuestionReference
参考
- — trace backend + pup exact flags, the runner contract (+ Go, export mode), polling + the false-negative sanity check, the trace-link scoping fix, limitations. Read before pup / the runner.
references/details.md - — making a deployed-only app locally runnable (step 3.5). Read when that gap shows.
references/local-setup.md - — the Python runner to copy + fill.
scripts/replay_runner_template.py
- ——trace后端 + pup具体参数,运行器契约(+ Go、导出模式),轮询+假阴性检查,trace链接范围修复,限制条件。在使用pup/运行器前阅读。
references/details.md - ——使仅部署的应用可在本地运行(步骤3.5)。当出现此需求时阅读。
references/local-setup.md - ——可复制并填充的Python运行器模板。
scripts/replay_runner_template.py