observability-driven-testing

Compare original and translation side by side

🇺🇸

Original

English

🇨🇳

Translation

Chinese

<objective> Production is the richest source of test design input: every error log, slow trace, and latency spike tells you where tests are missing. This skill closes the feedback loop between production observability and test creation, and makes trace structure a test assertion. A `200 OK` that silently hit the database on a path meant to be cache-only passes an HTTP assertion — a trace assertion catches it. Output: instrumented test runners, trace-based assertions, and a production-error-to-test pipeline. </objective>

<objective> 生产环境是测试设计输入的最丰富来源：每一条错误日志、缓慢的追踪记录、延迟峰值都能告诉你哪里缺少测试。本技能打通了生产可观测性与测试创建之间的反馈闭环，并将追踪结构作为测试断言。一个在本应走缓存的路径上静默访问数据库的`200 OK`请求能通过HTTP断言，但基于追踪的断言会捕捉到这个问题。输出结果：带有 instrumentation 的测试运行器、基于追踪的断言，以及从生产错误到测试的流水线。 </objective>

Quick Route

快速指引

Situation	Go to
Make test execution emit traces correlated with the app	OTel test-runner setup ( `references/trace-assertions.md` )
Assert which services were called / no error spans / latency	Traces as Test Evidence
Turn a Sentry/Datadog error into a test	Production Error to Test Pipeline
Decide which endpoints need tests next	Telemetry-Driven Test Prioritization
A trace assertion is flaky or a span never arrives	Failure Modes

场景	参考内容
让测试执行生成与应用关联的追踪记录	OTel测试运行器设置（ `references/trace-assertions.md` ）
断言哪些服务被调用/无错误 span/延迟	将追踪作为测试证据
将Sentry/Datadog错误转化为测试	生产错误转测试流水线
决定哪些端点需要优先测试	遥测驱动的测试优先级排序
基于追踪的断言不稳定或span始终未到达	故障模式

Discovery Questions

探索问题

Check

.agents/qa-project-context.md

first. If it exists, use it as context and skip questions already answered there.

Observability stack:

What APM/tracing tool is in place? (Datadog, New Relic, Honeycomb, Splunk Observability/SignalFx, ServiceNow Cloud Observability — formerly Lightstep, Dash0, Jaeger, Grafana Tempo, OpenTelemetry-native) — determines how you pull traces and which query syntax the diagnosis workflow uses.
Is OpenTelemetry instrumented in the application, and which services? — un-instrumented services are invisible and untestable via traces.
What logging infrastructure exists? (ELK, Loki, CloudWatch, Datadog Logs) — sets where log-by-trace-ID correlation happens.
Are structured logs used, or free-form text? — structured logs are parseable into test gaps; free-form needs a fingerprinting step first.

Tracing maturity:

Are distributed traces available across service boundaries? — without them, only single-service span assertions are possible.
What is the trace sampling rate? (100%, 10%, head-based, tail-based) — probabilistic sampling will randomly drop the trace a test asserts on; you must force-sample test traffic (see Failure Modes).
Can you search traces by error status, latency threshold, or custom attributes?
Are traces correlated with logs and metrics? — enables exemplars (metric → representative trace ID), which makes prioritization concrete.

Production error tracking:

What error tracking tool is used? (Sentry, SmartBear Insight Hub — formerly Bugsnag, Rollbar, Datadog Error Tracking, LaunchDarkly Observability — incl. session replay, formerly Highlight.io)
How are production errors triaged? (Automated, manual, ignored)
Is there a process for turning production errors into test cases?
What was the last production error that a test should have caught?

Test infrastructure:

Can tests emit telemetry? (Traces, custom metrics, structured logs)
Are test results correlated with application telemetry?
Do you have a test-to-code coverage mapping? — required to compute the error-rate-to-coverage matrix below.

首先查看

.agents/qa-project-context.md

。如果该文件存在，将其作为上下文，跳过已回答的问题。

可观测性栈：

当前使用的APM/追踪工具是什么？（Datadog、New Relic、Honeycomb、Splunk Observability/SignalFx、ServiceNow Cloud Observability——前身为Lightstep、Dash0、Jaeger、Grafana Tempo、原生OpenTelemetry）——决定了如何拉取追踪记录以及诊断工作流使用的查询语法。
应用中是否已接入OpenTelemetry instrumentation，涉及哪些服务？——未接入的服务无法通过追踪观测和测试。
当前使用的日志基础设施是什么？（ELK、Loki、CloudWatch、Datadog Logs）——决定了按trace ID关联日志的位置。
使用的是结构化日志还是自由文本日志？——结构化日志可解析为测试缺口；自由文本日志需要先进行指纹识别。

追踪成熟度：

是否支持跨服务边界的分布式追踪？——没有分布式追踪的话，只能对单服务的span进行断言。
追踪采样率是多少？（100%、10%、基于头部、基于尾部）——概率采样会随机丢弃测试断言依赖的追踪记录；必须强制采样测试流量（参见故障模式）。
是否能按错误状态、延迟阈值或自定义属性搜索追踪记录？
追踪记录是否与日志和指标关联？——支持示例（指标→代表性trace ID），让优先级排序更具体。

生产错误追踪：

使用的错误追踪工具是什么？（Sentry、SmartBear Insight Hub——前身为Bugsnag、Rollbar、Datadog Error Tracking、LaunchDarkly Observability——包含会话重放，前身为Highlight.io）
生产错误如何分类处理？（自动化、手动、忽略）
是否有将生产错误转化为测试用例的流程？
最近一次本应被测试捕捉到的生产错误是什么？

测试基础设施：

测试是否能生成遥测数据？（追踪记录、自定义指标、结构化日志）
测试结果是否与应用遥测数据关联？
是否有测试到代码覆盖率的映射？——这是计算下文错误率-覆盖率矩阵的必要条件。

Core Principles

核心原则

1. Production data informs test priorities

1. 生产数据指导测试优先级

The most valuable tests prevent real production errors — not theoretical edge cases, not contrived scenarios. Production error logs are a pre-prioritized backlog of tests you should have written, ordered by what real users actually hit.

最有价值的测试能预防真实的生产错误——而非理论上的边缘情况或人为构造的场景。生产错误日志是一份已预先排序的测试待办清单，按照真实用户实际遇到的问题排序。

2. Traces are test evidence

2. 追踪是测试证据

"The API returned 200" proves the endpoint responded. "The request hit the cache, skipped the database, and returned in <50ms" proves the system behaved correctly at every layer. Traces make tests deeper without making them more brittle.

「API返回200」只能证明端点有响应。「请求命中缓存、跳过数据库、且在50ms内返回」才能证明系统在每一层的行为都正确。基于追踪的测试能让测试更深入，同时不会增加测试的脆弱性。

3. Observability gaps equal test gaps

3. 可观测性缺口等于测试缺口

A code path with no traces, no logs, and no metrics is invisible — untestable in production and unverifiable during incidents. Observability coverage and test coverage are two views of the same problem.

没有追踪、日志和指标的代码路径是不可见的——在生产环境中无法测试，发生故障时也无法验证。可观测性覆盖和测试覆盖是同一问题的两个视角。

4. Close the feedback loop

4. 打通反馈闭环

The complete cycle: error detected → analyzed → test written → deployed → recurrence prevented. If your team finds production errors but does not systematically create tests, the same class of error recurs.

完整流程：错误检测→分析→编写测试→部署→防止复发。如果团队发现生产错误但没有系统性地创建测试，同类错误会再次发生。

Traces as Test Evidence

将追踪作为测试证据

Pin
@opentelemetry/semantic-conventions
to an exact version and treat sem-conv bumps as breaking. Trace assertions reference attribute names by string; those names drift across releases and your assertions silently break. v1.41.0 (April 2026) shipped GenAI breaking changes and a
process.executable
entity split, and moved
graphql.document
from Recommended to Opt-In. Pin the literal version and bump deliberately:
json
// package.json — exact pin, no caret
"@opentelemetry/semantic-conventions": "1.41.1"
bash
npm install --save-exact @opentelemetry/semantic-conventions@1.41.1
Do not introduce new OpenTracing shims. The OTel spec deprecated OpenTracing compatibility in March 2026 (removal no earlier than March 2027); new instrumentation should target native OTel APIs and OTLP.

Three patterns, all in

references/trace-assertions.md

OpenTelemetry integration in test infrastructure — instrument the test runner (
```
test-setup/tracing.ts
```
) so test execution correlates with application traces via
```
service.name
```
,
```
test.suite
```
, and
```
test.run_id
```
resource attributes. Flush from the runner's global teardown with an awaited
```
sdk.shutdown()
```
— not
```
process.on('beforeExit')
```
, which drops trailing spans.
Trace-based assertions — assert on trace structure, span attributes, and timing (which services were called, no ERROR spans, root-span latency, DB operations) instead of only the HTTP status. For unit-level span checks, use an in-process InMemorySpanExporter
+
SimpleSpanProcessor
and read
```
getFinishedSpans()
```
synchronously — no network, no
```
waitForTrace
```
, no timeout flake. Reserve the real collector +
```
waitForTrace
```
path for cross-process traces.
Distributed trace validation across services — an
```
assertTraceStructure
```
helper that verifies a request flowed through the expected services in order, with per-span attribute and
```
maxDuration
```
checks.

For declarative trace-based assertions (YAML/UI-driven instead of hand-rolled span queries), the OSS Tracetest project (

kubeshop/tracetest

) is still available, but the last public OSS release is v1.7.1 (Oct 2024) with low recent activity — evaluate maintenance before adopting. Tracetest's commercial Cloud offering was end-of-lifed October 2024; do not set up Tracetest Cloud, users will hit a dead product.

将
@opentelemetry/semantic-conventions
固定到精确版本，并将sem-conv版本升级视为破坏性变更。基于追踪的断言通过字符串引用属性名称；这些名称会在不同版本中变化，导致断言无声失效。2026年4月发布的v1.41.0版本引入了GenAI破坏性变更、拆分了
process.executable
实体，并将
graphql.document
从推荐项移至可选项。请固定具体版本并谨慎升级：
json
// package.json — 精确固定，不使用 caret 符号
"@opentelemetry/semantic-conventions": "1.41.1"
bash
npm install --save-exact @opentelemetry/semantic-conventions@1.41.1
不要引入新的OpenTracing垫片。 OTel规范在2026年3月弃用了OpenTracing兼容性（移除时间不早于2027年3月）；新的 instrumentation 应针对原生OTel API和OTLP。

三种模式，均在

references/trace-assertions.md

中：

测试基础设施中的OpenTelemetry集成——为测试运行器添加 instrumentation（
```
test-setup/tracing.ts
```
），通过
```
service.name
```
、
```
test.suite
```
和
```
test.run_id
```
资源属性让测试执行与应用追踪记录关联。在运行器的全局清理阶段通过等待
```
sdk.shutdown()
```
完成刷新——不要使用
```
process.on('beforeExit')
```
，它会丢弃末尾的span。
基于追踪的断言——针对追踪结构、span属性和时序（哪些服务被调用、无ERROR span、根span延迟、数据库操作）进行断言，而非仅检查HTTP状态。对于单元级span检查，使用进程内的**
```
InMemorySpanExporter
```
+
```
SimpleSpanProcessor
```
**，同步读取
```
getFinishedSpans()
```
——无需网络、无需
```
waitForTrace
```
、无超时不稳定问题。仅在跨进程追踪时使用真实收集器+
```
waitForTrace
```
方案。
跨服务分布式追踪验证——一个
```
assertTraceStructure
```
工具函数，用于验证请求是否按预期顺序流经各个服务，并检查每个span的属性和
```
maxDuration
```
。

对于声明式的基于追踪的断言（基于YAML/UI而非手动编写span查询），开源项目Tracetest（

kubeshop/tracetest

）仍可用，但最新的公开开源版本是2024年10月的v1.7.1，近期活跃度较低——在采用前需评估维护情况。Tracetest的商业云服务已于2024年10月终止；请勿搭建Tracetest Cloud，用户会遇到已停止服务的产品。

Log-Informed Test Design

日志驱动的测试设计

Analyze production error logs for test gaps

分析生产错误日志以发现测试缺口

Production errors are the highest-priority input for test creation. Each unhandled error is a missing test. See

references/log-and-error-pipeline.md

for the

analyze-production-errors.ts

script that maps each production error to test coverage, assigns a priority by frequency and recency, and suggests a test layer (unit/integration/e2e) from the error characteristics.

生产错误是测试创建的最高优先级输入。每一个未处理的错误都对应一个缺失的测试。请查看

references/log-and-error-pipeline.md

中的

analyze-production-errors.ts

脚本，该脚本将每个生产错误映射到测试覆盖率，根据频率和时效性分配优先级，并根据错误特征建议测试层级（单元/集成/E2E）。

Categorize errors: covered vs. uncovered

错误分类：已覆盖 vs 未覆盖

1. Export production errors from error tracker (Sentry, Insight Hub, etc.)
   - Filter: last 30 days, count > 5 (ignore one-off errors)
   - Group by: error message fingerprint

2. For each error group:
   a. Does a test exist that would catch this error?
      → Yes: the test is either not running or has a gap (investigate)
      → No: this is a test gap (create a test)

   b. What layer should the test live at?
      → TypeError, null reference → unit test
      → Timeout, connection error → integration test with fault injection
      → UI rendering error → E2E test
      → Data inconsistency → contract test or database test

3. Output: prioritized list of tests to create, ordered by:
   error frequency × user impact × recency

1. 从错误追踪工具（Sentry、Insight Hub等）导出生产错误
   - 筛选条件：最近30天，出现次数>5（忽略一次性错误）
   - 分组依据：错误消息指纹

2. 针对每个错误组：
   a. 是否存在能捕捉该错误的测试？
      → 是：测试未运行或存在缺口（需调查）
      → 否：这是测试缺口（需创建测试）

   b. 测试应属于哪个层级？
      → TypeError、空引用 → 单元测试
      → 超时、连接错误 → 带故障注入的集成测试
      → UI渲染错误 → E2E测试
      → 数据不一致 → 契约测试或数据库测试

3. 输出：按以下顺序排序的待创建测试优先级列表：
   错误频率 × 用户影响 × 时效性

Prioritize test creation by error frequency and impact

根据错误频率和影响确定测试创建优先级

Prioritize using a 2×2 of frequency (high/low) vs. impact (high/low): P0 = high-frequency + high-impact (fix now), P1 = low-frequency + high-impact (next sprint), P2 = high-frequency + low-impact (this sprint), P3 = both low (backlog). Impact indicators: high = payment/auth failure, data loss, crash; low = UI glitch, slow-but-functional response.

使用频率（高/低）vs影响（高/低）的2×2矩阵进行优先级排序：P0=高频率+高影响（立即修复），P1=低频率+高影响（下一个迭代），P2=高频率+低影响（当前迭代），P3=两者均低（待办清单）。影响判断指标：高=支付/认证失败、数据丢失、崩溃；低=UI小故障、响应缓慢但可用。

Telemetry-Driven Test Prioritization

遥测驱动的测试优先级排序

Score endpoints by error-weighted gap

按错误加权缺口为端点评分

Invest test effort proportional to real usage and real failure. Gap Score is the canonical formula used throughout this skill:

Gap Score = (error_rate × requests_per_day) / max(test_count, 1)

This is error-weighted: it ranks an endpoint by the absolute volume of failing requests it produces, divided by how much test coverage already guards it. (If you instead want a volume-weighted lens that surfaces high-traffic-but-healthy endpoints, multiply by

(1 + error_rate)

rather than

error_rate

— a different question, not the matrix's labels.)

根据真实使用情况和真实失败情况分配测试资源。缺口分数是本技能中使用的标准公式：

缺口分数 = (错误率 × 每日请求数) / max(测试数量, 1)

这是错误加权的：它根据端点产生的失败请求绝对数量进行排名，再除以已有的测试覆盖率。（如果想要一个基于流量加权的视角，突出高流量但健康的端点，可以将

错误率

替换为

(1 + 错误率)

——这是不同的需求，而非当前矩阵的用途。）

Error rate by endpoint to test coverage mapping

端点错误率与测试覆盖率映射

Endpoint           | Requests/day | Error Rate | Test Count | Gap Score
POST /api/orders   | 50,000       | 0.3%       | 2          | 75   CRITICAL
PUT  /api/profile  | 5,000        | 1.2%       | 1          | 60   CRITICAL
DELETE /api/items  | 2,000        | 0.8%       | 0          | 16   HIGH
POST /api/auth     | 80,000       | 0.1%       | 8          | 10   OK
GET  /api/search   | 200,000      | 0.05%      | 15         | 6.7  OK

Gap Score = (error_rate × requests_per_day) / max(test_count, 1)
Labels: CRITICAL ≥ 50, HIGH 12–49, OK < 12.

Action: create tests for endpoints at HIGH or above, highest score first.

Every label above is derived from the formula and the stated thresholds — copy the formula and you reproduce the matrix exactly. Pick your own thresholds, but state them; never hand-label.

Exemplars close the metric → trace → test loop. When a high-error endpoint surfaces in this matrix, OTel exemplars let you jump straight from the error-rate metric to a representative failing trace ID, then walk that trace (below) to write the test — instead of hunting for a matching trace by hand.

端点           | 每日请求数 | 错误率 | 测试数量 | 缺口分数
POST /api/orders   | 50,000       | 0.3%       | 2          | 75   CRITICAL
PUT  /api/profile  | 5,000        | 1.2%       | 1          | 60   CRITICAL
DELETE /api/items  | 2,000        | 0.8%       | 0          | 16   HIGH
POST /api/auth     | 80,000       | 0.1%       | 8          | 10   OK
GET  /api/search   | 200,000      | 0.05%      | 15         | 6.7  OK

缺口分数 = (错误率 × 每日请求数) / max(测试数量, 1)
标签：CRITICAL ≥ 50，HIGH 12–49，OK < 12。

行动：为HIGH及以上级别的端点创建测试，按分数从高到低排序。

以上所有标签均由公式和指定阈值得出——复制公式即可完全重现该矩阵。你可以自定义阈值，但需明确说明；切勿手动标注。

示例打通了指标→追踪→测试的闭环。 当高错误率端点出现在该矩阵中时，OTel 示例允许你直接从错误率指标跳转到代表性失败请求的trace ID，然后通过该追踪记录（如下）编写测试——无需手动查找匹配的追踪记录。

Hot path analysis

热路径分析

Identify the most-traversed code paths in production and ensure they have proportional test coverage.

1. Extract top 20 endpoints by request volume from APM data
2. For each endpoint, trace the code path through services
3. Map each service-level span to test coverage data
4. Identify hot paths with zero or low test coverage

Output:
  /api/checkout → cart-service → pricing-service → payment-service
  Coverage: cart-service (82%) → pricing-service (45%) → payment-service (91%)
  Gap: pricing-service discount calculation has 45% coverage on a critical path
  Action: Add tests for discount edge cases in pricing-service

Pair endpoint-level traffic data with continuous profiling to find CPU and allocation hot paths inside endpoints, not just at the boundary. The OTel profiling signal entered public alpha on 2026-03-26 (OTLP path
/v1development/profiles
), with GA targeted for Q3 2026 — treat it as not-yet-production. Production-ready alternatives today: Pyroscope, Parca, Polar Signals, Datadog Profiling. eBPF zero-instrumentation profilers (no SDK changes): Polar Signals, Parca, Grafana Beyla.

Zero-instrumentation observability — when adding the OTel SDK isn't feasible, eBPF tools capture HTTP/gRPC traces from kernel syscalls without code changes: Beyla (Grafana), Cilium Tetragon, Pixie, Coroot. Useful for legacy or polyglot services where SDK rollout takes quarters.

OTel Weaver generates type-safe instrumentation code from semantic-convention YAML — keeping trace assertions in sync with sem-conv bumps. Worth adopting if you maintain custom conventions or hit attribute drift between versions.

识别生产环境中流量最大的代码路径，并确保它们有相应的测试覆盖率。

1. 从APM数据中提取流量前20的端点
2. 针对每个端点，追踪其跨服务的代码路径
3. 将每个服务级span映射到测试覆盖率数据
4. 识别零覆盖率或低覆盖率的热路径

输出：
  /api/checkout → cart-service → pricing-service → payment-service
  覆盖率：cart-service (82%) → pricing-service (45%) → payment-service (91%)
  缺口：pricing-service的折扣计算在关键路径上仅45%覆盖率
  行动：为pricing-service的折扣边缘情况添加测试

将端点级流量数据与持续分析结合，以发现端点内部的CPU和内存分配热路径，而不仅仅是边界。OTel 分析信号于2026-03-26进入公开alpha阶段（OTLP路径
/v1development/profiles
），预计2026年第三季度正式发布——目前暂不建议用于生产环境。当前可用的生产就绪替代方案：Pyroscope、Parca、Polar Signals、Datadog Profiling。无需修改SDK的eBPF零 instrumentation 分析器：Polar Signals、Parca、Grafana Beyla。

零 instrumentation 可观测性——当无法添加OTel SDK时，eBPF工具可从内核系统调用中捕获HTTP/gRPC追踪记录，无需修改代码：Beyla（Grafana）、Cilium Tetragon、Pixie、Coroot。适用于遗留系统或多语言服务，这些场景下SDK部署可能需要数月时间。

OTel Weaver从语义规范YAML生成类型安全的instrumentation代码——确保基于追踪的断言与sem-conv版本升级保持同步。如果你维护自定义规范或遇到版本间属性变化的问题，值得采用。

Production Error to Test Pipeline

生产错误转测试流水线

The most important workflow in this skill: turning production errors into tests that prevent recurrence.

1. ERROR DETECTED
   Source: Sentry, Datadog, CloudWatch, or any error tracker
   Capture: error message, stack trace, request context, trace ID, user impact

2. REPRODUCE
   - Pull the trace from the observability platform (exemplar → trace ID if available)
   - Identify the exact request parameters and state that triggered the error
   - Reproduce locally or in staging with equivalent input
   - If not reproducible: add targeted logging and wait for recurrence

3. WRITE TEST
   - Choose the right layer (unit for logic bugs, integration for service interactions)
   - Test must fail before the fix (red-green verification)
   - Document the originating production error in the test name or a comment

4. FIX AND DEPLOY
   - Fix the bug; verify the test passes with the fix
   - Deploy fix + test together

5. VERIFY ELIMINATION
   - Monitor the same error in production after deploy
   - Confirm error count drops to zero
   - If it recurs: the fix was incomplete, repeat from step 2

See

references/log-and-error-pipeline.md

for a full test built from Sentry issue PROJ-4521 (null shipping address → null reference), asserting either a

(whichever your contract uses) at the API layer plus the E2E checkout prompt. The test name and a comment document the originating error, frequency, and context — the convention to follow when creating tests from production signals.

本技能中最重要的工作流：将生产错误转化为测试，防止错误复发。

1. 错误检测
   来源：Sentry、Datadog、CloudWatch或任何错误追踪工具
   捕获内容：错误消息、堆栈跟踪、请求上下文、trace ID、用户影响

2. 复现
   - 从可观测平台拉取追踪记录（如果可用，通过示例→trace ID）
   - 确定触发错误的精确请求参数和状态
   - 在本地或预发布环境中用等效输入复现
   - 若无法复现：添加针对性日志，等待错误再次发生

3. 编写测试
   - 选择合适的层级（逻辑错误用单元测试，服务交互用集成测试）
   - 修复前测试必须失败（红绿色验证）
   - 在测试名称或注释中记录原始生产错误

4. 修复并部署
   - 修复bug；验证测试在修复后通过
   - 同时部署修复代码和测试

5. 验证错误消除
   - 部署后监控生产环境中的同一错误
   - 确认错误计数降至零
   - 若错误复发：修复不完整，从步骤2重复

请查看

references/log-and-error-pipeline.md

中的完整测试示例，该示例基于Sentry问题PROJ-4521（空配送地址→空引用），在API层断言返回

或

（取决于你的契约），并包含E2E结账流程的测试。测试名称和注释记录了原始错误、频率和上下文——这是从生产信号创建测试时应遵循的规范。

Establish the team feedback loop

建立团队反馈闭环

Weekly error review (30 min): pull the top 10 new errors by frequency from the error tracker. For each: assign an owner, create a test, or mark as known/acceptable. An error tracker with thousands of unresolved entries that nobody reads is the anti-pattern.
Incident close gate: add "What test would have prevented this?" to every postmortem; the test is created (or the gap is explicitly recorded) before the incident is closed. Tie this to a checklist item so it is auditable, not aspirational.

每周错误评审（30分钟）：从错误追踪工具中提取出现频率最高的10个新错误。针对每个错误：分配负责人、创建测试，或标记为已知/可接受。错误追踪工具中有数千条未处理条目且无人查看的情况是反模式。
故障关闭检查项：在每个事后分析中添加「哪些测试可以预防该故障？」；在故障关闭前必须创建测试（或明确记录测试缺口）。将此作为检查项，确保可审计，而非仅停留在口头承诺。

Diagnosis Workflows

诊断工作流

Trace a failing request end-to-end

端到端追踪失败请求

When a test fails or a production error occurs, use the trace to understand exactly what happened.

1. Get the trace ID (from test output, error tracker, or user report)

2. Open the trace in your APM tool
   - Jaeger: /trace/{traceId}
   - Datadog: /apm/traces?traceId={traceId}
   - Honeycomb: query by trace.trace_id

3. Walk the span tree
   - Root span: what did the user request?
   - Child spans: which services were called?
   - Error spans: where did it fail? (which span FIRST shows an error)
   - Slow spans: where did latency accumulate?

4. Correlate with logs
   - Filter logs by trace ID to see every log entry for this request
   - Look for warnings or errors that precede the failure

5. Identify the root cause
   - Is the error in your code, a dependency, or infrastructure?
   - Transient failure or persistent bug?

当测试失败或生产错误发生时，使用追踪记录准确了解问题。

1. 获取trace ID（从测试输出、错误追踪工具或用户报告中）

2. 在APM工具中打开追踪记录
   - Jaeger：/trace/{traceId}
   - Datadog：/apm/traces?traceId={traceId}
   - Honeycomb：按trace.trace_id查询

3. 遍历span树
   - 根span：用户请求的是什么？
   - 子span：调用了哪些服务？
   - 错误span：哪里失败了？（哪个span首次出现错误）
   - 慢span：延迟出现在哪里？

4. 关联日志
   - 按trace ID筛选日志，查看该请求的所有日志条目
   - 查找故障发生前的警告或错误

5. 确定根本原因
   - 错误出现在你的代码、依赖项还是基础设施中？
   - 是临时故障还是持续存在的bug？

Correlate test failures with production telemetry

将测试失败与生产遥测数据关联

When a test fails, query your observability platform: (1) search production errors for matching messages (last 7 days); (2) search traces for the same HTTP route with ERROR status. Matches exist → the bug is real and affecting users, prioritize the fix. No matches → likely a test-only issue or a new bug not yet in production. This turns "probably flaky" into "confirmed production impact" or "test-only issue."

当测试失败时，查询可观测平台：(1) 搜索最近7天的生产错误中是否有匹配的消息；(2) 搜索同一HTTP路由带有ERROR状态的追踪记录。存在匹配项→bug真实存在且影响用户，优先修复。无匹配项→可能是仅测试环境存在的问题或尚未进入生产环境的新bug。这能将「可能不稳定」转化为「已确认生产影响」或「仅测试环境问题」。

Anti-Patterns

反模式

1. Ignoring production signals

1. 忽略生产信号

The error tracker has 500 unresolved errors nobody looks at; the suite passes, so the team assumes quality is fine. Fix: run the weekly 30-minute error review above — top 10 new errors, each assigned an owner, a test, or a known/acceptable mark.

错误追踪工具中有500条未处理错误但无人查看；测试套件通过，团队就认为质量没问题。修复方案：运行上述每周30分钟的错误评审——处理前10个新错误，为每个错误分配负责人、创建测试，或标记为已知/可接受。

2. Testing only what is easy to observe

2. 仅测试易于观测的内容

Teams assert HTTP status and response time while ignoring data consistency, background-job completion, and cache coherence. Fix: add spans to background jobs, cache ops, and async workflows, then assert on them. If it runs in production, it should produce telemetry.

团队只断言HTTP状态和响应时间，而忽略数据一致性、后台作业完成情况和缓存一致性。修复方案：为后台作业、缓存操作和异步工作流添加span，然后对其进行断言。任何在生产环境运行的内容都应生成遥测数据。

3. No feedback loop between production and testing

3. 生产与测试之间无反馈闭环

SRE handles errors, QA writes tests, neither shares systematically, the same class of bug recurs. Fix: establish the production-error-to-test pipeline and the incident-close gate.

SRE处理错误，QA编写测试，双方无系统性共享，同类bug重复发生。修复方案：建立生产错误转测试流水线和故障关闭检查项。

4. Over-instrumenting tests without acting on data

4. 过度为测试添加instrumentation但不分析数据

Thousands of metrics and logs emitted, nobody analyzes them — cost with no benefit. Fix: start with three specific questions you want test telemetry to answer; build those dashboards; add instrumentation only when you have a new question.

生成数千个指标和日志，但无人分析——只有成本没有收益。修复方案：从三个你希望测试遥测数据回答的具体问题开始；构建对应的仪表盘；仅当有新问题时才添加instrumentation。

5. Using traces only for debugging, not for assertions

5. 仅将追踪用于调试，而非断言

Traces treated as a post-break debugging tool rather than a source of assertions that prevent breaks. Fix: add trace-based assertions to integration tests — correct services called, efficient queries, expected cache hits. These catch regressions HTTP-level assertions miss.

将追踪视为故障后的调试工具，而非用于预防故障的断言来源。修复方案：为集成测试添加基于追踪的断言——验证调用了正确的服务、查询高效、命中预期缓存。这些断言能捕捉HTTP级断言遗漏的回归问题。

6. Asserting against a probabilistically sampled trace

6. 针对概率采样的追踪记录进行断言

Head/probabilistic sampling randomly drops the trace the test is asserting on, producing intermittent failures. Fix: force-sample test traffic (

OTEL_TRACES_SAMPLER=always_on

or a per-request override) so every asserted trace is recorded.

头部/概率采样会随机丢弃测试断言依赖的追踪记录，导致间歇性失败。修复方案：强制采样测试流量（

OTEL_TRACES_SAMPLER=always_on

或按请求覆盖），确保每个被断言的追踪记录都被记录。

Failure Modes

故障模式

Symptom	Likely cause	Fix or check
`waitForTrace` times out, span never arrives	Sampling dropped it, or exporter didn't flush before assert	Set `OTEL_TRACES_SAMPLER=always_on` for the test run; flush via awaited `sdk.shutdown()` in global teardown
Trailing spans from the last test missing	Flushed from `process.on('beforeExit')` (doesn't fire on exit/signal)	Move shutdown to the runner's global teardown hook; `await sdk.shutdown()`
App spans not part of the test's trace	`traceparent` header not propagated by the app	Confirm the app reads/forwards W3C `traceparent` ; check the OTel propagator is configured
Assertion on `db.system` / `graphql.document` /GenAI attrs suddenly fails	sem-conv version bump renamed/moved the attribute	Pin `@opentelemetry/semantic-conventions` exact; diff the release notes; update assertions deliberately
No spans reach the collector in CI	`OTEL_EXPORTER_ENDPOINT` unreachable from the CI network	Point at the in-CI collector address; smoke-test with the Verification step below
`status?.code === 'ERROR'` matches nothing despite real errors	Collector serializes status as `2` / `'STATUS_CODE_ERROR'` , not `'ERROR'`	Match what your collector actually emits (see note in `references/trace-assertions.md` )

症状	可能原因	修复或检查方法
`waitForTrace` 超时，span始终未到达	采样丢弃了该记录，或断言前导出器未完成刷新	测试运行时设置 `OTEL_TRACES_SAMPLER=always_on` ；在全局清理阶段通过等待 `sdk.shutdown()` 完成刷新
上一个测试的末尾span丢失	使用 `process.on('beforeExit')` 进行刷新（退出/信号触发时不会执行）	将刷新操作移至运行器的全局清理钩子； `await sdk.shutdown()`
应用span未纳入测试的追踪记录	应用未传播 `traceparent` 头	确认应用读取/转发W3C `traceparent` ；检查OTEL传播器配置
针对 `db.system` / `graphql.document` /GenAI属性的断言突然失败	sem-conv版本升级重命名/移动了属性	精确固定 `@opentelemetry/semantic-conventions` 版本；对比版本说明；针对性更新断言
CI中无span到达收集器	`OTEL_EXPORTER_ENDPOINT` 无法从CI网络访问	指向CI内部的收集器地址；使用下文的验证步骤进行冒烟测试
尽管存在真实错误，但 `status?.code === 'ERROR'` 无匹配结果	收集器将状态序列化为 `2` / `'STATUS_CODE_ERROR'` ，而非 `'ERROR'`	匹配收集器实际输出的内容（参见 `references/trace-assertions.md` 中的说明）

Verification

验证

Prove the telemetry path works before trusting any trace assertion, smallest first:

bash

undefined

在信任任何基于追踪的断言之前，先证明遥测路径可行，从最小的测试开始：