dt-obs-genai

Compare original and translation side by side

🇺🇸

Original

English
🇨🇳

Translation

Chinese

AI Observability (GenAI) Skill

AI Observability(GenAI)技能

Analyze AI Observability signals from customer GenAI applications using DQL — golden signals, LLM signals, token and cost analytics (with usage attribution and prompt-caching economics), agent signals (including loop/runaway detection and Smartscape topology), conversation/session-level analytics, guardrails, and evaluation quality.

使用DQL分析客户GenAI应用的AI可观测性信号——包括黄金信号、LLM信号、令牌与成本分析(含使用归因和提示缓存经济性)、Agent信号(含循环/失控检测和Smartscape拓扑)、对话/会话级分析、防护机制以及评估质量。

Core Capabilities

核心功能

Golden Signals

黄金信号

The four classic observability signals — traffic, errors, latency, and saturation — apply directly to GenAI applications. Traffic is request throughput over time; errors are spans where
span.status_code == "error"
; latency is the
duration
field (a Grail duration value — divide by the
1ms
literal,
duration / 1ms
, for a numeric millisecond value); saturation is proxied by total token throughput per minute (input + output tokens combined).
dql
fetch spans, from: now()-24h
| filter isNotNull(gen_ai.request.model)
| summarize total = count(), errors = countIf(span.status_code == "error"), by: {gen_ai.request.model}
| fieldsAdd error_rate_pct = if(total > 0, errors * 100.0 / total, else: 0.0)
| sort error_rate_pct desc
Full traffic, latency, and saturation queries: See references/golden-signals.md
四大经典可观测性信号——流量、错误、延迟、饱和度——直接适用于GenAI应用。流量指随时间变化的请求吞吐量;错误指
span.status_code == "error"
的spans;延迟为
duration
字段(Grail时长值——除以
1ms
字面量
duration / 1ms
可得到毫秒数值);饱和度由每分钟总令牌吞吐量(输入+输出令牌总和)间接体现。
dql
fetch spans, from: now()-24h
| filter isNotNull(gen_ai.request.model)
| summarize total = count(), errors = countIf(span.status_code == "error"), by: {gen_ai.request.model}
| fieldsAdd error_rate_pct = if(total > 0, errors * 100.0 / total, else: 0.0)
| sort error_rate_pct desc
完整的流量、延迟和饱和度查询: 参见references/golden-signals.md

LLM Signals

LLM信号

LLM signals describe which model and provider served each request, what operation type was invoked (
chat
,
execute_tool
,
invoke_agent
,
create_agent
), and how tokens were consumed. Use these to benchmark provider latency, compare model performance, and understand the token distribution across model-provider combinations.
dql
fetch spans, from: now()-24h
| filter isNotNull(gen_ai.request.model)
| summarize p95_ms = percentile(duration, 95) / 1ms, requests = count(), by: {gen_ai.provider.name}
| sort p95_ms desc
Slowest models, token usage by model: See references/llm-signals.md
LLM信号描述每个请求由哪个模型和提供商提供服务、调用的操作类型(
chat
execute_tool
invoke_agent
create_agent
)以及令牌的消耗方式。利用这些信号可以基准测试提供商延迟、比较模型性能,并了解令牌在不同模型-提供商组合中的分布情况。
dql
fetch spans, from: now()-24h
| filter isNotNull(gen_ai.request.model)
| summarize p95_ms = percentile(duration, 95) / 1ms, requests = count(), by: {gen_ai.provider.name}
| sort p95_ms desc
最慢模型、按模型统计令牌使用量: 参见references/llm-signals.md

Cost and Tokens

成本与令牌

Token consumption is the primary cost driver. Dynatrace stores
gen_ai.usage.input_tokens
and
gen_ai.usage.output_tokens
on every span — there is no stored cost field; estimated cost must be derived by multiplying token sums by the per-model price you supply. Use these queries to identify the highest-spend model-provider combinations and detect token-burn spikes.
dql
fetch spans, from: now()-24h
| filter isNotNull(gen_ai.usage.input_tokens) or isNotNull(gen_ai.usage.output_tokens)
| summarize input_tokens = sum(gen_ai.usage.input_tokens), output_tokens = sum(gen_ai.usage.output_tokens), total_tokens = sum(gen_ai.usage.input_tokens) + sum(gen_ai.usage.output_tokens), by: {gen_ai.provider.name, gen_ai.request.model}
| sort total_tokens desc
Token spikes, cost estimation, most expensive prompts, usage attribution, prompt-caching economics: See references/cost-and-tokens.md
令牌消耗是主要的成本驱动因素。Dynatrace在每个span上存储
gen_ai.usage.input_tokens
gen_ai.usage.output_tokens
——数据中没有存储成本字段;必须通过将令牌总和乘以您提供的单模型价格来估算成本。使用这些查询可以识别成本最高的模型-提供商组合,并检测令牌消耗峰值。
dql
fetch spans, from: now()-24h
| filter isNotNull(gen_ai.usage.input_tokens) or isNotNull(gen_ai.usage.output_tokens)
| summarize input_tokens = sum(gen_ai.usage.input_tokens), output_tokens = sum(gen_ai.usage.output_tokens), total_tokens = sum(gen_ai.usage.input_tokens) + sum(gen_ai.usage.output_tokens), by: {gen_ai.provider.name, gen_ai.request.model}
| sort total_tokens desc
令牌峰值、成本估算、最昂贵提示、使用归因、提示缓存经济性: 参见references/cost-and-tokens.md

Agent Signals

Agent信号

GenAI agents emit spans for each tool invocation (
execute_tool
), agent step (
invoke_agent
), and agent creation (
create_agent
). Use agent signals to identify which tools are called most often and which agents are failing. For structural questions — which agents, models, and providers exist and how they connect — query the GenAI Smartscape entities (
GENAI_AGENT
,
GENAI_MODEL
,
GENAI_PROVIDER
,
GENAI_SERVICE
) instead of scanning spans; this is a feature-flag-gated preview.
dql
fetch spans, from: now()-24h
| filter isNotNull(gen_ai.agent.name)
| summarize total = count(), errors = countIf(span.status_code == "error"), by: {gen_ai.agent.name}
| fieldsAdd error_rate_pct = if(total > 0, errors * 100.0 / total, else: 0.0)
| sort errors desc
Tool usage, failing agents, agent step latency, loop/runaway detection, Smartscape topology: See references/agent-signals.md
GenAI Agent会为每个工具调用(
execute_tool
)、Agent步骤(
invoke_agent
)和Agent创建(
create_agent
)生成spans。利用Agent信号可以识别调用最频繁的工具以及失败的Agent。对于结构性问题——存在哪些Agent、模型和提供商以及它们之间的连接方式——请查询GenAI Smartscape实体(
GENAI_AGENT
GENAI_MODEL
GENAI_PROVIDER
GENAI_SERVICE
)而非扫描spans;这是一个受功能标志控制的预览功能。
dql
fetch spans, from: now()-24h
| filter isNotNull(gen_ai.agent.name)
| summarize total = count(), errors = countIf(span.status_code == "error"), by: {gen_ai.agent.name}
| fieldsAdd error_rate_pct = if(total > 0, errors * 100.0 / total, else: 0.0)
| sort errors desc
工具使用情况、失败Agent、Agent步骤延迟、循环/失控检测、Smartscape拓扑: 参见references/agent-signals.md

Conversation Analytics

对话分析

Per-span and per-trace signals measure one request or one turn. When the application propagates
gen_ai.conversation.id
, you can roll spans up to the session level — cost per conversation, how deep conversations run, and which sessions are runaway-expensive or error-prone. This is the unit that matters for chargeback and user-perceived reliability.
dql
fetch spans, from: now()-24h
| filter isNotNull(gen_ai.conversation.id)
| summarize turns = countDistinct(trace.id), total_tokens = sum(gen_ai.usage.input_tokens) + sum(gen_ai.usage.output_tokens), errors = countIf(span.status_code == "error"), by: {gen_ai.conversation.id}
| sort total_tokens desc
Cost/depth per conversation, session error rate: See references/conversation-analytics.md
单span和单追踪信号衡量的是一次请求或一轮对话。当应用传播
gen_ai.conversation.id
时,您可以将spans汇总到会话级别——每次对话的成本、对话的深度以及哪些会话成本过高或容易出错。这是计费和用户感知可靠性的关键单元。
dql
fetch spans, from: now()-24h
| filter isNotNull(gen_ai.conversation.id)
| summarize turns = countDistinct(trace.id), total_tokens = sum(gen_ai.usage.input_tokens) + sum(gen_ai.usage.output_tokens), errors = countIf(span.status_code == "error"), by: {gen_ai.conversation.id}
| sort total_tokens desc
每次对话的成本/深度、会话错误率: 参见references/conversation-analytics.md

Guardrails

防护机制

Guardrails surface as
gen_ai.response.finish_reasons
on the span —
content_filter
means a safety filter blocked or redacted output,
length
means the response was truncated at the token limit — and as the proactive safety evaluators (
prompt-injection
,
pii-leakage
,
toxicity
,
bias
) in the evaluation bizevents. Use these to quantify blocked and truncated responses and tie them back to the LLM-judge safety verdicts.
dql
fetch spans, from: now()-24h
| filter isNotNull(gen_ai.response.finish_reasons)
| fieldsAdd finish_reason = gen_ai.response.finish_reasons
| expand finish_reason
| summarize calls = count(), by: {finish_reason, gen_ai.request.model}
| sort calls desc
Blocked (content filter), truncated (length), finish-reason breakdown: See references/guardrails.md
防护机制在span上表现为
gen_ai.response.finish_reasons
——
content_filter
表示安全过滤器拦截或编辑了输出,
length
表示响应在令牌限制处被截断——同时在评估bizevents中表现为主动安全评估器(
prompt-injection
pii-leakage
toxicity
bias
)。利用这些可以量化被拦截和截断的响应,并将其与LLM评判的安全结论关联起来。
dql
fetch spans, from: now()-24h
| filter isNotNull(gen_ai.response.finish_reasons)
| fieldsAdd finish_reason = gen_ai.response.finish_reasons
| expand finish_reason
| summarize calls = count(), by: {finish_reason, gen_ai.request.model}
| sort calls desc
被拦截(内容过滤器)、被截断(长度)、结束原因细分: 参见references/guardrails.md

Evaluation Quality

评估质量

Evaluation results are captured as bizevents (not spans) with
event.type == "gen_ai.evaluation.result"
. Each evaluator emits one bizevent per response, carrying the score, pass/fail label, explanation, and the exact Q&A pair. Use evaluation queries to monitor quality dimensions and surface failed responses with the LLM judge's reasoning. Each bizevent also carries the
trace.id
of the run that produced the evaluated response, so you can pivot from a quality failure to the spans that caused it.
dql
fetch bizevents, from: now()-24h
| filter event.type == "gen_ai.evaluation.result"
| filter gen_ai.evaluation.score.label == "fail"
| fields timestamp, gen_ai.evaluation.name, gen_ai.evaluation.score.value, gen_ai.evaluation.explanation, gen_ai.evaluation.input.question, gen_ai.evaluation.input.answer
| sort timestamp desc
Quality scores, failed evaluations, fail rates: See references/evaluations.md
评估结果以bizevents(而非spans)形式捕获,其中
event.type == "gen_ai.evaluation.result"
。每个评估器会为每个响应生成一个bizevent,包含分数、通过/失败标签、解释以及确切的问答对。使用评估查询可以监控质量维度,并展示带有LLM评判推理的失败响应。每个bizevent还包含生成被评估响应的运行的
trace.id
,因此您可以从质量失败转向导致失败的spans。
dql
fetch bizevents, from: now()-24h
| filter event.type == "gen_ai.evaluation.result"
| filter gen_ai.evaluation.score.label == "fail"
| fields timestamp, gen_ai.evaluation.name, gen_ai.evaluation.score.value, gen_ai.evaluation.explanation, gen_ai.evaluation.input.question, gen_ai.evaluation.input.answer
| sort timestamp desc
质量分数、失败评估、失败率: 参见references/evaluations.md

Empty-State Check

空状态检查

When any signal query returns no rows, do not report "no data found" — first confirm whether the application sends GenAI telemetry at all. These two presence checks show which signal families are present:
dql
fetch spans, from: now()-24h
| summarize 
    has_genai = countIf(isNotNull(gen_ai.request.model)), 
    has_tokens = countIf(isNotNull(gen_ai.usage.input_tokens) or isNotNull(gen_ai.usage.output_tokens)), 
    has_agents = countIf(isNotNull(gen_ai.agent.name)), 
    has_tools = countIf(gen_ai.operation.name == "execute_tool"), 
    has_conversation = countIf(isNotNull(gen_ai.conversation.id)), 
    has_finish_reason = countIf(isNotNull(gen_ai.response.finish_reasons)), 
    has_cached_tokens = countIf(isNotNull(gen_ai.usage.cache_read.input_tokens) or isNotNull(gen_ai.usage.cache_creation.input_tokens)),
    total = count()
    
dql
fetch bizevents, from: now()-24h
| filter event.type == "gen_ai.evaluation.result"
| summarize evals = count()
If
has_genai
is zero, report that the application appears not to be instrumented for AI Observability yet — not "no data found". If
has_genai
is non-zero but a specific family (
has_tokens
,
has_agents
,
has_tools
,
has_conversation
,
has_finish_reason
,
has_cached_tokens
,
evals
) is zero, only that signal type is missing — for example
has_conversation == 0
means session-level analytics are unavailable because the app does not propagate a conversation id, and
has_cached_tokens == 0
means prompt-caching telemetry is not being reported. These optional families may use different attribute names depending on the provider/SDK; verify before reporting them absent.

当任何信号查询返回无数据行时,请勿报告“未找到数据”——首先确认应用是否发送GenAI遥测数据。以下两个存在性检查会显示哪些信号类别存在:
dql
fetch spans, from: now()-24h
| summarize 
    has_genai = countIf(isNotNull(gen_ai.request.model)), 
    has_tokens = countIf(isNotNull(gen_ai.usage.input_tokens) or isNotNull(gen_ai.usage.output_tokens)), 
    has_agents = countIf(isNotNull(gen_ai.agent.name)), 
    has_tools = countIf(gen_ai.operation.name == "execute_tool"), 
    has_conversation = countIf(isNotNull(gen_ai.conversation.id)), 
    has_finish_reason = countIf(isNotNull(gen_ai.response.finish_reasons)), 
    has_cached_tokens = countIf(isNotNull(gen_ai.usage.cache_read.input_tokens) or isNotNull(gen_ai.usage.cache_creation.input_tokens)),
    total = count()
    
dql
fetch bizevents, from: now()-24h
| filter event.type == "gen_ai.evaluation.result"
| summarize evals = count()
如果
has_genai
为零,请报告该应用似乎尚未针对AI可观测性进行 instrumentation——而非“未找到数据”。如果
has_genai
非零但某个特定类别(
has_tokens
has_agents
has_tools
has_conversation
has_finish_reason
has_cached_tokens
evals
)为零,则仅该信号类型缺失——例如
has_conversation == 0
表示会话级分析不可用,因为应用未传播对话ID;
has_cached_tokens == 0
表示未报告提示缓存遥测数据。这些可选类别可能根据提供商/SDK使用不同的属性名称;在报告缺失之前请先验证。

Agent Instructions

Agent指令

Act First, Refine Later

先行动,后优化

When a user asks for analysis, proceed immediately with sensible defaults. Do not ask for parameter values you can reasonably assume.
Default values when not specified:
ParameterDefaultRationale
TimeframeLast 24 h (
from: now()-24h
)
Covers a full operational day without being too narrow
Model scopeAll models (no model filter)Shows the full picture; user can narrow after seeing results
Provider scopeAll providersSame rationale as model scope
Token thresholdNoneShow all — let the data reveal the outliers
Exception — cost prices. Per-model prices are the one input you cannot default (there is no cost field in the data). Ask the user for them before estimating USD; never use prices from memory. See cost-and-tokens.md.
当用户要求分析时,请立即使用合理的默认值执行。不要询问您可以合理假设的参数值。
未指定时的默认值:
参数默认值理由
时间范围过去24小时 (
from: now()-24h
)
涵盖完整的运营日,同时不会过于狭窄
模型范围所有模型(无模型过滤器)展示完整情况;用户查看结果后可进一步缩小范围
提供商范围所有提供商与模型范围的理由相同
令牌阈值显示所有数据——让数据揭示异常值
例外情况——成本价格。 单模型价格是您无法默认的唯一输入(数据中没有成本字段)。在估算美元成本之前,请先向用户询问价格;切勿使用记忆中的价格。参见cost-and-tokens.md

Empty-State Rule

空状态规则

When any signal query returns no rows, run the two presence checks in the Empty-State Check capability above before responding — never reply "no data found". If
has_genai
is zero, report that the application appears not to be instrumented for AI Observability yet; if only a specific family is zero, say which signal type is missing.
当任何信号查询返回无数据行时,请先运行上述空状态检查中的两个存在性检查,再进行响应——切勿回复“未找到数据”。如果
has_genai
为零,请报告该应用似乎尚未针对AI可观测性进行instrumentation;如果仅某个特定类别为零,请说明缺失的信号类型。

Scope Boundary

范围边界

This skill covers AI Observability signals for customer GenAI applications only. Product documentation and configuration how-to questions (e.g., "How do I configure the Dynatrace OTLP endpoint?") go to
ask-dynatrace-docs
— this skill does not contain product configuration how-tos.
本技能仅涵盖客户GenAI应用的AI可观测性信号。产品文档和配置操作类问题(例如“如何配置Dynatrace OTLP端点?”)请转至
ask-dynatrace-docs
——本技能不包含产品配置操作指南。

Understanding User Intent

理解用户意图

Map user requests and prompt-starter phrasings to capabilities:
User Request / Prompt StarterCapabilityReference File
"Understand AI Observability signals"All signal categories overviewThis SKILL.md
"Analyze LLM latency and errors", "LLM errors", "error rate by model"Golden Signalsgolden-signals.md
"Which models are slowest right now?", "compare latency across providers"LLM Signalsllm-signals.md
"Show token usage by model", "token usage spikes"Cost and Tokenscost-and-tokens.md
"Break down cost by model and provider", "which prompts are most expensive?"Cost and Tokenscost-and-tokens.md
"Trace a failing agent run", "show failed tool calls"Agent Signalsagent-signals.md
"Break down agent steps by latency"Agent Signalsagent-signals.md
"Map agent topology", "which models does this agent use?", "list GenAI agents/models/providers"Agent Signals (Smartscape)agent-signals.md
"Is an agent stuck in a loop?", "find runaway agents", "what caused the token spike?"Agent Signals (loops)agent-signals.md
"Cost per conversation", "most expensive sessions", "how deep do conversations run?"Conversation Analyticsconversation-analytics.md
"Stitch together an agent trajectory", "filter by session id", "connect traces across a session"Conversation Analyticsconversation-analytics.md
"How often are responses blocked/filtered?", "are responses being truncated?", "finish reasons"Guardrailsguardrails.md
"Cost by application/user/tenant", "who is driving token spend?"Cost and Tokens (attribution)cost-and-tokens.md
"Do I have prompt caching?", "cache hit rate", "caching savings"Cost and Tokens (caching)cost-and-tokens.md
"Summarize evaluation quality scores", "show low-scoring responses", "show failed evaluations"Evaluation Qualityevaluations.md
"What signals am I missing?", "why is there no data?"Empty-State CheckThis SKILL.md

将用户请求和提示起始语映射到功能:
用户请求 / 提示起始语功能参考文件
"了解AI可观测性信号"所有信号类别概述本SKILL.md
"分析LLM延迟和错误"、"LLM错误"、"按模型统计错误率"黄金信号golden-signals.md
"目前哪些模型最慢?"、"比较不同提供商的延迟"LLM信号llm-signals.md
"按模型显示令牌使用量"、"令牌使用量峰值"成本与令牌cost-and-tokens.md
"按模型和提供商细分成本"、"哪些提示最昂贵?"成本与令牌cost-and-tokens.md
"追踪失败的Agent运行"、"显示失败的工具调用"Agent信号agent-signals.md
"按延迟细分Agent步骤"Agent信号agent-signals.md
"映射Agent拓扑"、"该Agent使用哪些模型?"、"列出GenAI Agents/模型/提供商"Agent信号(Smartscape)agent-signals.md
"Agent是否陷入循环?"、"查找失控Agent"、"令牌峰值的原因是什么?"Agent信号(循环)agent-signals.md
"每次对话的成本"、"最昂贵的会话"、"对话的深度如何?"对话分析conversation-analytics.md
"拼接Agent轨迹"、"按会话ID过滤"、"跨会话连接追踪"对话分析conversation-analytics.md
"响应被拦截/过滤的频率如何?"、"响应是否被截断?"、"结束原因"防护机制guardrails.md
"按应用/用户/租户统计成本"、"谁在驱动令牌消耗?"成本与令牌(归因)cost-and-tokens.md
"我是否启用了提示缓存?"、"缓存命中率"、"缓存节省"成本与令牌(缓存)cost-and-tokens.md
"总结评估质量分数"、"显示低分响应"、"显示失败评估"评估质量evaluations.md
"我缺少哪些信号?"、"为什么没有数据?"空状态检查本SKILL.md

Common Workflows

常见工作流

Workflow: Cost Investigation

工作流:成本调查

1. Run token usage by model and provider (cost-and-tokens.md → "Token usage by model and provider")
2. Identify the top model-provider combinations by total_tokens
3. For the top offenders, run token usage spikes to check for abnormal time windows
4. Use the cost-estimation template in "Most expensive prompts and models" to estimate USD spend — ask the user for per-model prices first (see "Exception — cost prices" under Agent Instructions)
5. Check for prompt-size outliers: high input_tokens / output_tokens ratio indicates large context windows
6. Attribute spend to a consumer (cost-and-tokens.md → "Usage attribution") and check whether prompt caching is enabled and effective (cost-and-tokens.md → "Prompt caching economics")
1. 运行按模型和提供商统计令牌使用量(cost-and-tokens.md → "按模型和提供商统计令牌使用量")
2. 确定总令牌数最高的模型-提供商组合
3. 针对排名靠前的组合,运行令牌使用量峰值查询以检查异常时间窗口
4. 使用"最昂贵的提示和模型"中的成本估算模板估算美元支出——首先向用户询问单模型价格(参见Agent指令下的"例外情况——成本价格")
5. 检查提示大小异常值:高输入令牌/输出令牌比率表示上下文窗口较大
6. 将支出归因于消费者(cost-and-tokens.md → "使用归因"),并检查提示缓存是否已启用且有效(cost-and-tokens.md → "提示缓存经济性")

Workflow: Token-Spike / Runaway Investigation

工作流:令牌峰值/失控调查

1. Run token usage spikes (cost-and-tokens.md → "Token usage spikes") to find the abnormal time window
2. Within that window, run repeated-tool-calls and runaway-turn queries (agent-signals.md → "Agent loops and runaway detection")
3. For a flagged trace.id, open the trace to see what the agent looped on
4. If conversation ids are present, re-run the loop query grouped by gen_ai.conversation.id to catch cross-turn loops (conversation-analytics.md)
1. 运行令牌使用量峰值查询(cost-and-tokens.md → "令牌使用量峰值")以找到异常时间窗口
2. 在该窗口内,运行重复工具调用和失控轮次查询(agent-signals.md → "Agent循环和失控检测")
3. 对于标记的trace.id,打开追踪查看Agent循环的内容
4. 如果存在对话ID,重新运行按gen_ai.conversation.id分组的循环查询以捕获跨轮次循环(conversation-analytics.md)

Workflow: Failing Agent Run

工作流:失败的Agent运行

1. Run failing agent activity query (agent-signals.md → "Failing agent activity")
2. Sort by errors desc to find the most error-prone agent
3. Take the trace.id from a failing span and open in Dynatrace distributed-tracing view
4. Check agent steps by latency (agent-signals.md) to see which operation type is slowest
1. 运行失败Agent活动查询(agent-signals.md → "失败Agent活动")
2. 按错误数降序排序以找到最容易出错的Agent
3. 从失败的span中获取trace.id并在Dynatrace分布式追踪视图中打开
4. 按延迟检查Agent步骤(agent-signals.md)以查看哪种操作类型最慢

Workflow: Guardrail & Safety Review

工作流:防护机制与安全审查

1. Run the guardrails presence check (guardrails.md) to confirm finish reasons are recorded
2. Run the finish-reason breakdown, then the blocked (content_filter) and truncated (length) queries
3. Correlate content-filter spikes with the prompt-injection evaluator and truncation with answer-completeness failures (evaluations.md)
4. For a specific block or failure, take the trace.id and pivot to the originating spans (evaluations.md → "Correlating evaluations to traces")
1. 运行防护机制存在性检查(guardrails.md)以确认已记录结束原因
2. 运行结束原因细分查询,然后运行被拦截(content_filter)和被截断(length)查询
3. 将内容过滤器峰值与提示注入评估器关联,将截断与回答完整性失败关联(evaluations.md)
4. 对于特定的拦截或失败,获取其trace.id并转向原始spans(evaluations.md → "将评估与追踪关联")

Workflow: Evaluation Review

工作流:评估审查

1. Run evaluation quality scores (evaluations.md → "Evaluation quality scores") to rank evaluators by avg_score asc
2. Focus on the lowest-scoring evaluator
3. Run failed evaluations (evaluations.md → "Failed evaluations") to surface the exact Q&A pairs and LLM judge explanations
4. Use the "Fail rate by evaluator" query to see how many responses fail each evaluator and the share of total evaluations
5. To root-cause a specific failure, take its trace.id and pivot to the originating spans (evaluations.md → "Correlating evaluations to traces")

1. 运行评估质量分数查询(evaluations.md → "评估质量分数")以按平均分数升序排列评估器
2. 关注得分最低的评估器
3. 运行失败评估查询(evaluations.md → "失败评估")以展示确切的问答对和LLM评判解释
4. 使用"按评估器统计失败率"查询查看每个评估器有多少响应失败以及在总评估中的占比
5. 要根本原因分析特定失败,获取其trace.id并转向原始spans(evaluations.md → "将评估与追踪关联")

References

参考资料

  • references/golden-signals.md — traffic, errors, latency, saturation
  • references/llm-signals.md — slowest models, provider latency, token usage by model
  • references/cost-and-tokens.md — token usage, spikes, cost-estimation template, usage attribution, prompt-caching economics
  • references/agent-signals.md — tool usage, failing agents, step latency, loop/runaway detection, Smartscape agent topology (preview)
  • references/conversation-analytics.md — session-level cost, depth, and error rate (
    gen_ai.conversation.id
    )
  • references/guardrails.md — blocked (content filter) and truncated (length) responses via finish reasons
  • references/evaluations.md — quality scores, failed evals, fail-rate, trace correlation (bizevents)
  • references/golden-signals.md — 流量、错误、延迟、饱和度
  • references/llm-signals.md — 最慢模型、提供商延迟、按模型统计令牌使用量
  • references/cost-and-tokens.md — 令牌使用量、峰值、成本估算模板、使用归因、提示缓存经济性
  • references/agent-signals.md — 工具使用情况、失败Agent、步骤延迟、循环/失控检测、Smartscape Agent拓扑(预览)
  • references/conversation-analytics.md — 会话级成本、深度和错误率(
    gen_ai.conversation.id
  • references/guardrails.md — 通过结束原因统计被拦截(内容过滤器)和被截断(长度)的响应
  • references/evaluations.md — 质量分数、失败评估、失败率、追踪关联(bizevents)