dt-obs-genai
Compare original and translation side by side
🇺🇸
Original
English🇨🇳
Translation
ChineseAI Observability (GenAI) Skill
AI Observability(GenAI)技能
Analyze AI Observability signals from customer GenAI applications using DQL — golden
signals, LLM signals, token and cost analytics (with usage attribution and prompt-caching
economics), agent signals (including loop/runaway detection and Smartscape topology),
conversation/session-level analytics, guardrails, and evaluation quality.
使用DQL分析客户GenAI应用的AI可观测性信号——包括黄金信号、LLM信号、令牌与成本分析(含使用归因和提示缓存经济性)、Agent信号(含循环/失控检测和Smartscape拓扑)、对话/会话级分析、防护机制以及评估质量。
Core Capabilities
核心功能
Golden Signals
黄金信号
The four classic observability signals — traffic, errors, latency, and saturation — apply
directly to GenAI applications. Traffic is request throughput over time; errors are spans
where ; latency is the field (a Grail duration
value — divide by the literal, , for a numeric millisecond value);
saturation is proxied by total token throughput per minute (input + output tokens combined).
span.status_code == "error"duration1msduration / 1msdql
fetch spans, from: now()-24h
| filter isNotNull(gen_ai.request.model)
| summarize total = count(), errors = countIf(span.status_code == "error"), by: {gen_ai.request.model}
| fieldsAdd error_rate_pct = if(total > 0, errors * 100.0 / total, else: 0.0)
| sort error_rate_pct desc→ Full traffic, latency, and saturation queries: See references/golden-signals.md
四大经典可观测性信号——流量、错误、延迟、饱和度——直接适用于GenAI应用。流量指随时间变化的请求吞吐量;错误指的spans;延迟为字段(Grail时长值——除以字面量可得到毫秒数值);饱和度由每分钟总令牌吞吐量(输入+输出令牌总和)间接体现。
span.status_code == "error"duration1msduration / 1msdql
fetch spans, from: now()-24h
| filter isNotNull(gen_ai.request.model)
| summarize total = count(), errors = countIf(span.status_code == "error"), by: {gen_ai.request.model}
| fieldsAdd error_rate_pct = if(total > 0, errors * 100.0 / total, else: 0.0)
| sort error_rate_pct desc→ 完整的流量、延迟和饱和度查询: 参见references/golden-signals.md
LLM Signals
LLM信号
LLM signals describe which model and provider served each request, what operation type
was invoked (, , , ), and how tokens were
consumed. Use these to benchmark provider latency, compare model performance, and
understand the token distribution across model-provider combinations.
chatexecute_toolinvoke_agentcreate_agentdql
fetch spans, from: now()-24h
| filter isNotNull(gen_ai.request.model)
| summarize p95_ms = percentile(duration, 95) / 1ms, requests = count(), by: {gen_ai.provider.name}
| sort p95_ms desc→ Slowest models, token usage by model: See references/llm-signals.md
LLM信号描述每个请求由哪个模型和提供商提供服务、调用的操作类型(、、、)以及令牌的消耗方式。利用这些信号可以基准测试提供商延迟、比较模型性能,并了解令牌在不同模型-提供商组合中的分布情况。
chatexecute_toolinvoke_agentcreate_agentdql
fetch spans, from: now()-24h
| filter isNotNull(gen_ai.request.model)
| summarize p95_ms = percentile(duration, 95) / 1ms, requests = count(), by: {gen_ai.provider.name}
| sort p95_ms desc→ 最慢模型、按模型统计令牌使用量: 参见references/llm-signals.md
Cost and Tokens
成本与令牌
Token consumption is the primary cost driver. Dynatrace stores
and on every span — there is no stored cost field; estimated
cost must be derived by multiplying token sums by the per-model price you supply. Use these
queries to identify the highest-spend model-provider combinations and detect token-burn spikes.
gen_ai.usage.input_tokensgen_ai.usage.output_tokensdql
fetch spans, from: now()-24h
| filter isNotNull(gen_ai.usage.input_tokens) or isNotNull(gen_ai.usage.output_tokens)
| summarize input_tokens = sum(gen_ai.usage.input_tokens), output_tokens = sum(gen_ai.usage.output_tokens), total_tokens = sum(gen_ai.usage.input_tokens) + sum(gen_ai.usage.output_tokens), by: {gen_ai.provider.name, gen_ai.request.model}
| sort total_tokens desc→ Token spikes, cost estimation, most expensive prompts, usage attribution, prompt-caching economics: See references/cost-and-tokens.md
令牌消耗是主要的成本驱动因素。Dynatrace在每个span上存储和——数据中没有存储成本字段;必须通过将令牌总和乘以您提供的单模型价格来估算成本。使用这些查询可以识别成本最高的模型-提供商组合,并检测令牌消耗峰值。
gen_ai.usage.input_tokensgen_ai.usage.output_tokensdql
fetch spans, from: now()-24h
| filter isNotNull(gen_ai.usage.input_tokens) or isNotNull(gen_ai.usage.output_tokens)
| summarize input_tokens = sum(gen_ai.usage.input_tokens), output_tokens = sum(gen_ai.usage.output_tokens), total_tokens = sum(gen_ai.usage.input_tokens) + sum(gen_ai.usage.output_tokens), by: {gen_ai.provider.name, gen_ai.request.model}
| sort total_tokens desc→ 令牌峰值、成本估算、最昂贵提示、使用归因、提示缓存经济性: 参见references/cost-and-tokens.md
Agent Signals
Agent信号
GenAI agents emit spans for each tool invocation (), agent step
(), and agent creation (). Use agent signals to identify
which tools are called most often and which agents are failing. For structural
questions — which agents, models, and providers exist and how they connect — query the
GenAI Smartscape entities (, , , )
instead of scanning spans; this is a feature-flag-gated preview.
execute_toolinvoke_agentcreate_agentGENAI_AGENTGENAI_MODELGENAI_PROVIDERGENAI_SERVICEdql
fetch spans, from: now()-24h
| filter isNotNull(gen_ai.agent.name)
| summarize total = count(), errors = countIf(span.status_code == "error"), by: {gen_ai.agent.name}
| fieldsAdd error_rate_pct = if(total > 0, errors * 100.0 / total, else: 0.0)
| sort errors desc→ Tool usage, failing agents, agent step latency, loop/runaway detection, Smartscape topology: See references/agent-signals.md
GenAI Agent会为每个工具调用()、Agent步骤()和Agent创建()生成spans。利用Agent信号可以识别调用最频繁的工具以及失败的Agent。对于结构性问题——存在哪些Agent、模型和提供商以及它们之间的连接方式——请查询GenAI Smartscape实体(、、、)而非扫描spans;这是一个受功能标志控制的预览功能。
execute_toolinvoke_agentcreate_agentGENAI_AGENTGENAI_MODELGENAI_PROVIDERGENAI_SERVICEdql
fetch spans, from: now()-24h
| filter isNotNull(gen_ai.agent.name)
| summarize total = count(), errors = countIf(span.status_code == "error"), by: {gen_ai.agent.name}
| fieldsAdd error_rate_pct = if(total > 0, errors * 100.0 / total, else: 0.0)
| sort errors desc→ 工具使用情况、失败Agent、Agent步骤延迟、循环/失控检测、Smartscape拓扑: 参见references/agent-signals.md
Conversation Analytics
对话分析
Per-span and per-trace signals measure one request or one turn. When the application
propagates , you can roll spans up to the session level — cost
per conversation, how deep conversations run, and which sessions are runaway-expensive or
error-prone. This is the unit that matters for chargeback and user-perceived reliability.
gen_ai.conversation.iddql
fetch spans, from: now()-24h
| filter isNotNull(gen_ai.conversation.id)
| summarize turns = countDistinct(trace.id), total_tokens = sum(gen_ai.usage.input_tokens) + sum(gen_ai.usage.output_tokens), errors = countIf(span.status_code == "error"), by: {gen_ai.conversation.id}
| sort total_tokens desc→ Cost/depth per conversation, session error rate: See references/conversation-analytics.md
单span和单追踪信号衡量的是一次请求或一轮对话。当应用传播时,您可以将spans汇总到会话级别——每次对话的成本、对话的深度以及哪些会话成本过高或容易出错。这是计费和用户感知可靠性的关键单元。
gen_ai.conversation.iddql
fetch spans, from: now()-24h
| filter isNotNull(gen_ai.conversation.id)
| summarize turns = countDistinct(trace.id), total_tokens = sum(gen_ai.usage.input_tokens) + sum(gen_ai.usage.output_tokens), errors = countIf(span.status_code == "error"), by: {gen_ai.conversation.id}
| sort total_tokens desc→ 每次对话的成本/深度、会话错误率: 参见references/conversation-analytics.md
Guardrails
防护机制
Guardrails surface as on the span — means a
safety filter blocked or redacted output, means the response was truncated at the
token limit — and as the proactive safety evaluators (, ,
, ) in the evaluation bizevents. Use these to quantify blocked and truncated
responses and tie them back to the LLM-judge safety verdicts.
gen_ai.response.finish_reasonscontent_filterlengthprompt-injectionpii-leakagetoxicitybiasdql
fetch spans, from: now()-24h
| filter isNotNull(gen_ai.response.finish_reasons)
| fieldsAdd finish_reason = gen_ai.response.finish_reasons
| expand finish_reason
| summarize calls = count(), by: {finish_reason, gen_ai.request.model}
| sort calls desc→ Blocked (content filter), truncated (length), finish-reason breakdown: See references/guardrails.md
防护机制在span上表现为——表示安全过滤器拦截或编辑了输出,表示响应在令牌限制处被截断——同时在评估bizevents中表现为主动安全评估器(、、、)。利用这些可以量化被拦截和截断的响应,并将其与LLM评判的安全结论关联起来。
gen_ai.response.finish_reasonscontent_filterlengthprompt-injectionpii-leakagetoxicitybiasdql
fetch spans, from: now()-24h
| filter isNotNull(gen_ai.response.finish_reasons)
| fieldsAdd finish_reason = gen_ai.response.finish_reasons
| expand finish_reason
| summarize calls = count(), by: {finish_reason, gen_ai.request.model}
| sort calls desc→ 被拦截(内容过滤器)、被截断(长度)、结束原因细分: 参见references/guardrails.md
Evaluation Quality
评估质量
Evaluation results are captured as bizevents (not spans) with
. Each evaluator emits one bizevent per
response, carrying the score, pass/fail label, explanation, and the exact Q&A pair.
Use evaluation queries to monitor quality dimensions and surface failed responses
with the LLM judge's reasoning. Each bizevent also carries the of the run that
produced the evaluated response, so you can pivot from a quality failure to the spans that
caused it.
event.type == "gen_ai.evaluation.result"trace.iddql
fetch bizevents, from: now()-24h
| filter event.type == "gen_ai.evaluation.result"
| filter gen_ai.evaluation.score.label == "fail"
| fields timestamp, gen_ai.evaluation.name, gen_ai.evaluation.score.value, gen_ai.evaluation.explanation, gen_ai.evaluation.input.question, gen_ai.evaluation.input.answer
| sort timestamp desc→ Quality scores, failed evaluations, fail rates: See references/evaluations.md
评估结果以bizevents(而非spans)形式捕获,其中。每个评估器会为每个响应生成一个bizevent,包含分数、通过/失败标签、解释以及确切的问答对。使用评估查询可以监控质量维度,并展示带有LLM评判推理的失败响应。每个bizevent还包含生成被评估响应的运行的,因此您可以从质量失败转向导致失败的spans。
event.type == "gen_ai.evaluation.result"trace.iddql
fetch bizevents, from: now()-24h
| filter event.type == "gen_ai.evaluation.result"
| filter gen_ai.evaluation.score.label == "fail"
| fields timestamp, gen_ai.evaluation.name, gen_ai.evaluation.score.value, gen_ai.evaluation.explanation, gen_ai.evaluation.input.question, gen_ai.evaluation.input.answer
| sort timestamp desc→ 质量分数、失败评估、失败率: 参见references/evaluations.md
Empty-State Check
空状态检查
When any signal query returns no rows, do not report "no data found" — first confirm
whether the application sends GenAI telemetry at all. These two presence checks show
which signal families are present:
dql
fetch spans, from: now()-24h
| summarize
has_genai = countIf(isNotNull(gen_ai.request.model)),
has_tokens = countIf(isNotNull(gen_ai.usage.input_tokens) or isNotNull(gen_ai.usage.output_tokens)),
has_agents = countIf(isNotNull(gen_ai.agent.name)),
has_tools = countIf(gen_ai.operation.name == "execute_tool"),
has_conversation = countIf(isNotNull(gen_ai.conversation.id)),
has_finish_reason = countIf(isNotNull(gen_ai.response.finish_reasons)),
has_cached_tokens = countIf(isNotNull(gen_ai.usage.cache_read.input_tokens) or isNotNull(gen_ai.usage.cache_creation.input_tokens)),
total = count()
dql
fetch bizevents, from: now()-24h
| filter event.type == "gen_ai.evaluation.result"
| summarize evals = count()If is zero, report that the application appears not to be instrumented for AI
Observability yet — not "no data found". If is non-zero but a specific family
(, , , , ,
, ) is zero, only that signal type is missing — for example
means session-level analytics are unavailable because the app does
not propagate a conversation id, and means prompt-caching telemetry
is not being reported. These optional families may use different attribute names depending on
the provider/SDK; verify before reporting them absent.
has_genaihas_genaihas_tokenshas_agentshas_toolshas_conversationhas_finish_reasonhas_cached_tokensevalshas_conversation == 0has_cached_tokens == 0当任何信号查询返回无数据行时,请勿报告“未找到数据”——首先确认应用是否发送GenAI遥测数据。以下两个存在性检查会显示哪些信号类别存在:
dql
fetch spans, from: now()-24h
| summarize
has_genai = countIf(isNotNull(gen_ai.request.model)),
has_tokens = countIf(isNotNull(gen_ai.usage.input_tokens) or isNotNull(gen_ai.usage.output_tokens)),
has_agents = countIf(isNotNull(gen_ai.agent.name)),
has_tools = countIf(gen_ai.operation.name == "execute_tool"),
has_conversation = countIf(isNotNull(gen_ai.conversation.id)),
has_finish_reason = countIf(isNotNull(gen_ai.response.finish_reasons)),
has_cached_tokens = countIf(isNotNull(gen_ai.usage.cache_read.input_tokens) or isNotNull(gen_ai.usage.cache_creation.input_tokens)),
total = count()
dql
fetch bizevents, from: now()-24h
| filter event.type == "gen_ai.evaluation.result"
| summarize evals = count()如果为零,请报告该应用似乎尚未针对AI可观测性进行 instrumentation——而非“未找到数据”。如果非零但某个特定类别(、、、、、、)为零,则仅该信号类型缺失——例如表示会话级分析不可用,因为应用未传播对话ID;表示未报告提示缓存遥测数据。这些可选类别可能根据提供商/SDK使用不同的属性名称;在报告缺失之前请先验证。
has_genaihas_genaihas_tokenshas_agentshas_toolshas_conversationhas_finish_reasonhas_cached_tokensevalshas_conversation == 0has_cached_tokens == 0Agent Instructions
Agent指令
Act First, Refine Later
先行动,后优化
When a user asks for analysis, proceed immediately with sensible defaults. Do not ask
for parameter values you can reasonably assume.
Default values when not specified:
| Parameter | Default | Rationale |
|---|---|---|
| Timeframe | Last 24 h ( | Covers a full operational day without being too narrow |
| Model scope | All models (no model filter) | Shows the full picture; user can narrow after seeing results |
| Provider scope | All providers | Same rationale as model scope |
| Token threshold | None | Show all — let the data reveal the outliers |
Exception — cost prices. Per-model prices are the one input you cannot
default (there is no cost field in the data). Ask the user for them before
estimating USD; never use prices from memory. See cost-and-tokens.md.
当用户要求分析时,请立即使用合理的默认值执行。不要询问您可以合理假设的参数值。
未指定时的默认值:
| 参数 | 默认值 | 理由 |
|---|---|---|
| 时间范围 | 过去24小时 ( | 涵盖完整的运营日,同时不会过于狭窄 |
| 模型范围 | 所有模型(无模型过滤器) | 展示完整情况;用户查看结果后可进一步缩小范围 |
| 提供商范围 | 所有提供商 | 与模型范围的理由相同 |
| 令牌阈值 | 无 | 显示所有数据——让数据揭示异常值 |
例外情况——成本价格。 单模型价格是您无法默认的唯一输入(数据中没有成本字段)。在估算美元成本之前,请先向用户询问价格;切勿使用记忆中的价格。参见cost-and-tokens.md。
Empty-State Rule
空状态规则
When any signal query returns no rows, run the two presence checks in the
Empty-State Check capability above before responding — never reply "no data found".
If is zero, report that the application appears not to be instrumented for AI
Observability yet; if only a specific family is zero, say which signal type is missing.
has_genai当任何信号查询返回无数据行时,请先运行上述空状态检查中的两个存在性检查,再进行响应——切勿回复“未找到数据”。如果为零,请报告该应用似乎尚未针对AI可观测性进行instrumentation;如果仅某个特定类别为零,请说明缺失的信号类型。
has_genaiScope Boundary
范围边界
This skill covers AI Observability signals for customer GenAI applications only.
Product documentation and configuration how-to questions (e.g., "How do I configure
the Dynatrace OTLP endpoint?") go to — this skill does not
contain product configuration how-tos.
ask-dynatrace-docs本技能仅涵盖客户GenAI应用的AI可观测性信号。产品文档和配置操作类问题(例如“如何配置Dynatrace OTLP端点?”)请转至——本技能不包含产品配置操作指南。
ask-dynatrace-docsUnderstanding User Intent
理解用户意图
Map user requests and prompt-starter phrasings to capabilities:
| User Request / Prompt Starter | Capability | Reference File |
|---|---|---|
| "Understand AI Observability signals" | All signal categories overview | This SKILL.md |
| "Analyze LLM latency and errors", "LLM errors", "error rate by model" | Golden Signals | golden-signals.md |
| "Which models are slowest right now?", "compare latency across providers" | LLM Signals | llm-signals.md |
| "Show token usage by model", "token usage spikes" | Cost and Tokens | cost-and-tokens.md |
| "Break down cost by model and provider", "which prompts are most expensive?" | Cost and Tokens | cost-and-tokens.md |
| "Trace a failing agent run", "show failed tool calls" | Agent Signals | agent-signals.md |
| "Break down agent steps by latency" | Agent Signals | agent-signals.md |
| "Map agent topology", "which models does this agent use?", "list GenAI agents/models/providers" | Agent Signals (Smartscape) | agent-signals.md |
| "Is an agent stuck in a loop?", "find runaway agents", "what caused the token spike?" | Agent Signals (loops) | agent-signals.md |
| "Cost per conversation", "most expensive sessions", "how deep do conversations run?" | Conversation Analytics | conversation-analytics.md |
| "Stitch together an agent trajectory", "filter by session id", "connect traces across a session" | Conversation Analytics | conversation-analytics.md |
| "How often are responses blocked/filtered?", "are responses being truncated?", "finish reasons" | Guardrails | guardrails.md |
| "Cost by application/user/tenant", "who is driving token spend?" | Cost and Tokens (attribution) | cost-and-tokens.md |
| "Do I have prompt caching?", "cache hit rate", "caching savings" | Cost and Tokens (caching) | cost-and-tokens.md |
| "Summarize evaluation quality scores", "show low-scoring responses", "show failed evaluations" | Evaluation Quality | evaluations.md |
| "What signals am I missing?", "why is there no data?" | Empty-State Check | This SKILL.md |
将用户请求和提示起始语映射到功能:
| 用户请求 / 提示起始语 | 功能 | 参考文件 |
|---|---|---|
| "了解AI可观测性信号" | 所有信号类别概述 | 本SKILL.md |
| "分析LLM延迟和错误"、"LLM错误"、"按模型统计错误率" | 黄金信号 | golden-signals.md |
| "目前哪些模型最慢?"、"比较不同提供商的延迟" | LLM信号 | llm-signals.md |
| "按模型显示令牌使用量"、"令牌使用量峰值" | 成本与令牌 | cost-and-tokens.md |
| "按模型和提供商细分成本"、"哪些提示最昂贵?" | 成本与令牌 | cost-and-tokens.md |
| "追踪失败的Agent运行"、"显示失败的工具调用" | Agent信号 | agent-signals.md |
| "按延迟细分Agent步骤" | Agent信号 | agent-signals.md |
| "映射Agent拓扑"、"该Agent使用哪些模型?"、"列出GenAI Agents/模型/提供商" | Agent信号(Smartscape) | agent-signals.md |
| "Agent是否陷入循环?"、"查找失控Agent"、"令牌峰值的原因是什么?" | Agent信号(循环) | agent-signals.md |
| "每次对话的成本"、"最昂贵的会话"、"对话的深度如何?" | 对话分析 | conversation-analytics.md |
| "拼接Agent轨迹"、"按会话ID过滤"、"跨会话连接追踪" | 对话分析 | conversation-analytics.md |
| "响应被拦截/过滤的频率如何?"、"响应是否被截断?"、"结束原因" | 防护机制 | guardrails.md |
| "按应用/用户/租户统计成本"、"谁在驱动令牌消耗?" | 成本与令牌(归因) | cost-and-tokens.md |
| "我是否启用了提示缓存?"、"缓存命中率"、"缓存节省" | 成本与令牌(缓存) | cost-and-tokens.md |
| "总结评估质量分数"、"显示低分响应"、"显示失败评估" | 评估质量 | evaluations.md |
| "我缺少哪些信号?"、"为什么没有数据?" | 空状态检查 | 本SKILL.md |
Common Workflows
常见工作流
Workflow: Cost Investigation
工作流:成本调查
1. Run token usage by model and provider (cost-and-tokens.md → "Token usage by model and provider")
2. Identify the top model-provider combinations by total_tokens
3. For the top offenders, run token usage spikes to check for abnormal time windows
4. Use the cost-estimation template in "Most expensive prompts and models" to estimate USD spend — ask the user for per-model prices first (see "Exception — cost prices" under Agent Instructions)
5. Check for prompt-size outliers: high input_tokens / output_tokens ratio indicates large context windows
6. Attribute spend to a consumer (cost-and-tokens.md → "Usage attribution") and check whether prompt caching is enabled and effective (cost-and-tokens.md → "Prompt caching economics")1. 运行按模型和提供商统计令牌使用量(cost-and-tokens.md → "按模型和提供商统计令牌使用量")
2. 确定总令牌数最高的模型-提供商组合
3. 针对排名靠前的组合,运行令牌使用量峰值查询以检查异常时间窗口
4. 使用"最昂贵的提示和模型"中的成本估算模板估算美元支出——首先向用户询问单模型价格(参见Agent指令下的"例外情况——成本价格")
5. 检查提示大小异常值:高输入令牌/输出令牌比率表示上下文窗口较大
6. 将支出归因于消费者(cost-and-tokens.md → "使用归因"),并检查提示缓存是否已启用且有效(cost-and-tokens.md → "提示缓存经济性")Workflow: Token-Spike / Runaway Investigation
工作流:令牌峰值/失控调查
1. Run token usage spikes (cost-and-tokens.md → "Token usage spikes") to find the abnormal time window
2. Within that window, run repeated-tool-calls and runaway-turn queries (agent-signals.md → "Agent loops and runaway detection")
3. For a flagged trace.id, open the trace to see what the agent looped on
4. If conversation ids are present, re-run the loop query grouped by gen_ai.conversation.id to catch cross-turn loops (conversation-analytics.md)1. 运行令牌使用量峰值查询(cost-and-tokens.md → "令牌使用量峰值")以找到异常时间窗口
2. 在该窗口内,运行重复工具调用和失控轮次查询(agent-signals.md → "Agent循环和失控检测")
3. 对于标记的trace.id,打开追踪查看Agent循环的内容
4. 如果存在对话ID,重新运行按gen_ai.conversation.id分组的循环查询以捕获跨轮次循环(conversation-analytics.md)Workflow: Failing Agent Run
工作流:失败的Agent运行
1. Run failing agent activity query (agent-signals.md → "Failing agent activity")
2. Sort by errors desc to find the most error-prone agent
3. Take the trace.id from a failing span and open in Dynatrace distributed-tracing view
4. Check agent steps by latency (agent-signals.md) to see which operation type is slowest1. 运行失败Agent活动查询(agent-signals.md → "失败Agent活动")
2. 按错误数降序排序以找到最容易出错的Agent
3. 从失败的span中获取trace.id并在Dynatrace分布式追踪视图中打开
4. 按延迟检查Agent步骤(agent-signals.md)以查看哪种操作类型最慢Workflow: Guardrail & Safety Review
工作流:防护机制与安全审查
1. Run the guardrails presence check (guardrails.md) to confirm finish reasons are recorded
2. Run the finish-reason breakdown, then the blocked (content_filter) and truncated (length) queries
3. Correlate content-filter spikes with the prompt-injection evaluator and truncation with answer-completeness failures (evaluations.md)
4. For a specific block or failure, take the trace.id and pivot to the originating spans (evaluations.md → "Correlating evaluations to traces")1. 运行防护机制存在性检查(guardrails.md)以确认已记录结束原因
2. 运行结束原因细分查询,然后运行被拦截(content_filter)和被截断(length)查询
3. 将内容过滤器峰值与提示注入评估器关联,将截断与回答完整性失败关联(evaluations.md)
4. 对于特定的拦截或失败,获取其trace.id并转向原始spans(evaluations.md → "将评估与追踪关联")Workflow: Evaluation Review
工作流:评估审查
1. Run evaluation quality scores (evaluations.md → "Evaluation quality scores") to rank evaluators by avg_score asc
2. Focus on the lowest-scoring evaluator
3. Run failed evaluations (evaluations.md → "Failed evaluations") to surface the exact Q&A pairs and LLM judge explanations
4. Use the "Fail rate by evaluator" query to see how many responses fail each evaluator and the share of total evaluations
5. To root-cause a specific failure, take its trace.id and pivot to the originating spans (evaluations.md → "Correlating evaluations to traces")1. 运行评估质量分数查询(evaluations.md → "评估质量分数")以按平均分数升序排列评估器
2. 关注得分最低的评估器
3. 运行失败评估查询(evaluations.md → "失败评估")以展示确切的问答对和LLM评判解释
4. 使用"按评估器统计失败率"查询查看每个评估器有多少响应失败以及在总评估中的占比
5. 要根本原因分析特定失败,获取其trace.id并转向原始spans(evaluations.md → "将评估与追踪关联")References
参考资料
- references/golden-signals.md — traffic, errors, latency, saturation
- references/llm-signals.md — slowest models, provider latency, token usage by model
- references/cost-and-tokens.md — token usage, spikes, cost-estimation template, usage attribution, prompt-caching economics
- references/agent-signals.md — tool usage, failing agents, step latency, loop/runaway detection, Smartscape agent topology (preview)
- references/conversation-analytics.md — session-level cost, depth, and error rate ()
gen_ai.conversation.id - references/guardrails.md — blocked (content filter) and truncated (length) responses via finish reasons
- references/evaluations.md — quality scores, failed evals, fail-rate, trace correlation (bizevents)
- references/golden-signals.md — 流量、错误、延迟、饱和度
- references/llm-signals.md — 最慢模型、提供商延迟、按模型统计令牌使用量
- references/cost-and-tokens.md — 令牌使用量、峰值、成本估算模板、使用归因、提示缓存经济性
- references/agent-signals.md — 工具使用情况、失败Agent、步骤延迟、循环/失控检测、Smartscape Agent拓扑(预览)
- references/conversation-analytics.md — 会话级成本、深度和错误率()
gen_ai.conversation.id - references/guardrails.md — 通过结束原因统计被拦截(内容过滤器)和被截断(长度)的响应
- references/evaluations.md — 质量分数、失败评估、失败率、追踪关联(bizevents)