qa-metrics

Compare original and translation side by side

🇺🇸

Original

English

🇨🇳

Translation

Chinese

<objective> A suite with 4,000 tests where 800 are disabled and 200 are flaky looks healthy on a slide and lies in production. This skill defines the metrics that actually change behavior — each one with a formula, a target, an owner, and a concrete action when it goes red — so the dashboard becomes a feedback loop instead of decoration. </objective>

<objective> 一个包含4000个测试用例的套件，其中800个被禁用、200个不稳定，在演示文稿上看起来很健康，但在生产环境中却漏洞百出。本技能定义了真正能改变行为的指标——每个指标都配有计算公式、目标值、负责人，以及指标异常时的具体行动——让仪表盘成为反馈闭环，而非装饰。 </objective>

Quick Route

快速指引

Your situation	Metric to reach for	Section / reference
No metrics yet, where to start	code coverage, flakiness, defect escape	Core Principles + `references/rollout.md`
Bugs escaping despite high coverage	defect escape rate + mutation score	Coverage Metrics, Defect Metrics
Tests flaky, devs ignore CI	flakiness rate, pass rate	Test Health Metrics
Slow pipeline, stale feedback	suite duration, parallelism efficiency	Execution Metrics
Justifying automation spend	automation ROI	Process Metrics
Picking targets for our stage	company-stage table	Setting Realistic Targets
Leadership wants delivery + quality	DORA + escape rate	Engineering Quality Metrics (DORA)
Building the dashboard view	(go to `qa-dashboard` )	—

你的场景	应关注的指标	章节/参考文档
尚未跟踪任何指标，不知从何入手	代码覆盖率、测试不稳定率、缺陷逃逸率	核心原则 + `references/rollout.md`
覆盖率很高但仍有漏洞流入生产	缺陷逃逸率 + 变异分数	覆盖指标、缺陷指标
测试不稳定，开发人员忽略CI结果	不稳定率、通过率	测试健康度指标
流水线缓慢，反馈滞后	套件执行时长、并行效率	执行指标
需证明自动化投入的价值	自动化ROI	流程指标
为团队阶段设定目标值	公司阶段目标表	设置合理目标
管理层希望兼顾交付速度与质量	DORA指标 + 缺陷逃逸率	工程质量指标（DORA）
构建仪表盘视图	（请使用 `qa-dashboard` 技能）	—

Discovery Questions

探索性问题

Check

.agents/qa-project-context.md

first — if it exists, use it as the foundation and skip anything already answered there. Then:

首先查看

.agents/qa-project-context.md

——如果该文件存在，以此为基础，跳过已解答的问题。然后：

Current State

当前状态

What metrics are you tracking today? (Even "we glance at CI pass rate sometimes" counts.)
Where does your test data live? (CI system, coverage reports, bug tracker, spreadsheets, nowhere)
Do you have any dashboards already? Who looks at them, and how often?
What tooling runs your CI pipeline? (GitHub Actions, GitLab CI, Jenkins, CircleCI)

你目前正在跟踪哪些指标？（即使是“我们偶尔查看CI通过率”也算）
测试数据存储在哪里？（CI系统、覆盖率报告、缺陷跟踪工具、电子表格，还是无存储）
你是否已有仪表盘？谁会查看，查看频率如何？
你使用什么工具运行CI流水线？（GitHub Actions、GitLab CI、Jenkins、CircleCI）

Stakeholders

利益相关方

Who are the stakeholders for quality metrics? (engineering, product, leadership, customers)
What does each care about? Engineers want flakiness; leadership wants escape trends; product wants release confidence.
Who will own each metric? If nobody owns it, nobody acts on it.

质量指标的利益相关方有哪些？（工程团队、产品团队、管理层、客户）
每个利益相关方关心什么？工程师关注测试不稳定率；管理层关注缺陷逃逸趋势；产品团队关注发布信心。
每个指标的负责人是谁？如果无人负责，就无人会采取行动。

Quality Problems

质量问题

What quality problems need visibility? (regressions, slow pipelines, flaky tests, coverage gaps, incidents)
What broke in production recently? Would a metric have caught it earlier?
What decisions are you making without data today?

哪些质量问题需要可视化？（回归问题、缓慢流水线、不稳定测试、覆盖缺口、故障事件）
最近生产环境中出现了哪些故障？是否有指标可以提前发现？
你目前在没有数据支撑的情况下做出哪些决策？

Goals

目标

What does "healthy test suite" mean for your team?
Any compliance or contractual quality requirements? (SLAs, SOC2, ISO)
Quick wins vs. full observability — what is the appetite for metrics infrastructure?

对你的团队而言，“健康的测试套件”意味着什么？
是否有合规或合同规定的质量要求？（SLAs、SOC2、ISO）
快速见效 vs 全面可观测——团队对指标基础设施的接受度如何？

Core Principles

核心原则

1. Metrics Should Drive Action, Not Just Dashboards

1. 指标应驱动行动，而非仅用于仪表盘

A metric without an action plan is decoration. For every metric, define: what threshold triggers action, what the action is, and who takes it. If flakiness crosses 5%, the on-call engineer investigates the top 3 flaky tests that week. No ambiguity.

没有行动计划的指标只是装饰。对于每个指标，定义：触发行动的阈值、具体行动内容，以及执行行动的人员。例如，如果不稳定率超过5%，值班工程师需调查当周排名前三的不稳定测试用例。不能有模糊地带。

2. Leading vs Lagging — Track Both, Act on Leading

2. 领先指标与滞后指标——两者都要跟踪，但重点关注领先指标

Defect escape rate and MTTR are lagging: you learn after users were hurt. Flakiness, coverage delta, and skipped-test count are leading: they predict escapes before they happen. Lagging metrics tell leadership whether quality is moving; leading metrics are where engineers spend their daily attention because they can still change the outcome.

缺陷逃逸率和MTTR是滞后指标：只有在用户受到影响后你才会得知。不稳定率、覆盖率变化量和跳过测试用例数是领先指标：它们能在缺陷逃逸前做出预测。滞后指标告诉管理层质量是否在提升；领先指标是工程师日常关注的重点，因为他们仍能改变结果。

3. Trend Over Snapshot

3. 关注趋势而非快照

A single number is nearly useless. Coverage at 72% means nothing; coverage trending 68% → 72% over three sprints tells a story. Display metrics as time series and evaluate direction, not absolute position.

单一数字几乎毫无意义。覆盖率72%本身没有意义；但如果在三个迭代中从68%上升到72%，这就说明了问题。将指标展示为时间序列，评估变化方向，而非绝对数值。

4. Every Metric Needs a Target and an Owner

4. 每个指标都需要目标值和负责人

A target makes a metric actionable; an owner makes it accountable. Without both it becomes background noise. Set targets from your team's maturity (see the targets table) and assign owners who can actually move the number.

目标值让指标可落地；负责人让指标有人问责。缺少两者的指标只会成为背景噪音。根据团队成熟度设定目标值（见目标表），并将指标分配给实际能改变数值的负责人。

5. Vanity Metrics Waste Everyone's Time

5. 虚荣指标浪费所有人的时间

"We have 4,000 tests" sounds impressive until 800 are disabled and 200 flaky. Count what matters: tests that run, pass reliably, and catch real bugs. The ultimate metric is whether users hit bugs that hurt the business — work backward from there. Defect escape rate connects directly to user experience; lines of test code connects to nothing.

“我们有4000个测试用例”听起来很厉害，但其中800个被禁用、200个不稳定。要统计真正重要的内容：能运行、稳定通过、能发现真实漏洞的测试用例。最终指标是用户是否遇到影响业务的漏洞——从这个目标倒推。缺陷逃逸率直接关联用户体验；测试代码行数则毫无意义。

Essential QA Metrics

核心QA指标

Each metric: definition, formula, recommended target, why it matters, and the action when it goes red.

每个指标包含：定义、计算公式、推荐目标值、重要性，以及指标异常时的行动。

Test Coverage Metrics

测试覆盖指标

How much of our system is verified by automated tests? (leading)

我们的系统有多少部分被自动化测试验证？（领先指标）

Code Coverage Percentage

代码覆盖率

The percentage of code exercised by automated tests (line, branch, or statement level).

Line coverage   = (lines executed by tests / total executable lines) × 100
Branch coverage = (branches executed by tests / total branches) × 100

Targets: Line: 70-85% for app code. Branch: 60-75%. Critical paths (payments, auth): 90%+.

Why it matters: Coverage identifies blind spots — code never exercised by tests is where bugs hide undetected.

When it goes red: Coverage drops on PR: block merge or flag. Low in critical module: create targeted tasks. Plateaus: check for dead code vs. genuinely untested logic.

Warning: Coverage measures execution, not assertion quality. Pair it with mutation testing (StrykerJS v9.6+, mutmut) for a truer picture. Stryker's Vitest runner tracks recent Vitest releases — check the runner's peer-dependency before pinning a Vitest version. Use

incremental: true

in monorepo CI to keep mutation runs cheap. For deep coverage/mutation analysis, see

coverage-analysis

自动化测试执行到的代码占比（行、分支或语句级别）。

Line coverage   = (lines executed by tests / total executable lines) × 100
Branch coverage = (branches executed by tests / total branches) × 100

目标值： 应用代码行覆盖率：70-85%。分支覆盖率：60-75%。关键路径（支付、认证）：90%+。

重要性： 覆盖率能识别盲区——从未被测试执行过的代码是漏洞隐藏的地方。

指标异常时的行动： PR中覆盖率下降：阻止合并或标记。关键模块覆盖率低：创建针对性任务。覆盖率停滞：检查是否存在死代码或真正未测试的逻辑。

注意： 覆盖率衡量的是执行情况，而非断言质量。需搭配变异测试（StrykerJS v9.6+、mutmut）才能获得更真实的情况。Stryker的Vitest runner支持最新Vitest版本——在固定Vitest版本前，请检查runner的peer依赖。在单体仓库CI中使用

incremental: true

以降低变异测试的运行成本。如需深入分析覆盖率/变异情况，请查看

coverage-analysis

技能。

Requirement Coverage Percentage

需求覆盖率

Requirement coverage = (features with at least one test / total features) × 100

Targets: 100% for P0/P1 features, 80%+ for P2.

Why it matters: A feature can have zero tests even if surrounding code is well-covered. Use test tags (

@feature:checkout

@story:PROJ-1234

) for traceability.

Requirement coverage = (features with at least one test / total features) × 100

目标值： P0/P1功能100%覆盖，P2功能80%+覆盖。

重要性： 即使周边代码覆盖率很高，某个功能也可能没有任何测试用例。使用测试标签（

@feature:checkout

、

@story:PROJ-1234

）实现可追溯性。

Risk Coverage Percentage

风险覆盖率

Risk coverage = (high-risk areas with automated tests / total high-risk areas) × 100

Targets: 95%+ for high-risk areas. Maintain a risk register and cross-reference against coverage data per module.

Risk coverage = (high-risk areas with automated tests / total high-risk areas) × 100

目标值： 高风险区域95%+覆盖。维护风险登记册，并按模块与覆盖率数据交叉核对。

Test Health Metrics

测试健康度指标

Can we trust our test suite? (leading)

我们可以信任测试套件吗？（领先指标）

Flakiness Rate

不稳定率

The percentage of test runs producing inconsistent results without code changes.

Flakiness rate = (test runs with flaky results / total test runs) × 100

Targets: Acceptable: <2%. Warning: 2-5%. Critical: >5%.

Why it matters: Flaky tests erode trust. Once developers think "probably just flaky," they stop paying attention to results at all. Flakiness is the single biggest threat to a suite's credibility.

When it goes red: Quarantine flaky tests immediately. Investigate the top 3 weekly — most flakiness comes from a small number of tests. Common causes: timing/race conditions, shared state, external dependencies, order-dependent tests. Tests flaky for 30+ days should be deleted or rewritten.

Detection: Buildkite Test Analytics, Datadog Test Optimization (formerly Datadog CI Visibility — ships Flaky Test Management, Auto Test Retries, Early Flake Detection, Failed Test Replay, and Test Impact Analysis; its Bits AI Dev Agent can auto-open PRs to fix flaky tests), Trunk Flaky Tests, or a script comparing results across runs on the same commit.

无代码变更时，测试运行结果不一致的占比。

Flakiness rate = (test runs with flaky results / total test runs) × 100

目标值： 可接受：<2%。警告：2-5%。严重：>5%。

重要性： 不稳定测试会侵蚀信任。一旦开发人员觉得“可能只是不稳定”，他们就会完全不再关注测试结果。不稳定是测试套件可信度的最大威胁。

指标异常时的行动： 立即隔离不稳定测试用例。调查当周排名前三的不稳定测试——大多数不稳定问题来自少数测试用例。常见原因：时序/竞争条件、共享状态、外部依赖、测试顺序依赖。不稳定超过30天的测试用例应删除或重写。

检测工具： Buildkite Test Analytics、Datadog Test Optimization（原Datadog CI Visibility——提供不稳定测试管理、自动测试重试、早期不稳定检测、失败测试重放和测试影响分析；其Bits AI Dev Agent可自动创建PR修复不稳定测试）、Trunk Flaky Tests，或比较同一提交多次运行结果的脚本。

Pass Rate Trend (7-Day Rolling)

通过率趋势（7天滚动）

Pass rate = (green CI runs / total CI runs) × 100  [rolling 7 days]

Targets: Healthy: >95%. Warning: 90-95%. Broken: <90%.

The 7-day rolling average smooths daily noise and reveals the real trend. A consistently red build means developers ignore the pipeline.

Pass rate = (green CI runs / total CI runs) × 100  [rolling 7 days]

目标值： 健康：>95%。警告：90-95%。异常：<90%。

7天滚动平均值可平滑日常波动，反映真实趋势。持续失败的构建意味着开发人员会忽略流水线。

Disabled and Skipped Test Count

禁用和跳过的测试用例数

Total tests marked

skip

disabled

pending

xit

xdescribe

or equivalent.

Target: Trend toward zero. Skipped tests older than 2 sprints: fix or delete. Add a CI step that fails if skipped count exceeds 5% of total tests.

Why it matters: Skipped tests are invisible coverage gaps. A suite with 500 passing and 150 skipped tests has a

150 / (500 + 150) = 23%

gap that dashboards hide.

标记为

skip

、

disabled

、

pending

、

xit

、

xdescribe

或等效状态的测试用例总数。

目标值： 趋势趋近于零。跳过超过2个迭代的测试用例：修复或删除。添加CI步骤，如果跳过测试数超过总测试数的5%则失败。

重要性： 跳过的测试是隐藏的覆盖缺口。一个有500个通过测试、150个跳过测试的套件存在

150 / (500 + 150) = 23%

的缺口，而仪表盘会隐藏这个问题。

Test Suite Duration

测试套件执行时长

Wall-clock time from suite start to completion.

Targets: Unit: <5 min. Integration: <10 min. E2E: <15 min. Full pipeline: <30 min (the stage targets are the real budget; <30 min is the sum, not a separate looser bar).

Why it matters: Slow tests break the feedback loop. 45-minute results mean developers have already context-switched.

When it goes red: Profile slowest tests (10% often account for 50% of runtime). Increase parallelism. Move slow tests post-merge. Check for unnecessary setup/teardown. Plot duration over time and alert on step changes (>20% increase in a week).

从套件启动到完成的挂钟时间。

目标值： 单元测试：<5分钟。集成测试：<10分钟。E2E测试：<15分钟。完整流水线：<30分钟（阶段目标是实际预算；<30分钟是各阶段时长之和，而非单独的宽松标准）。

重要性： 缓慢的测试会打破反馈闭环。45分钟的结果意味着开发人员已经切换了工作上下文。

指标异常时的行动： 分析最慢的测试（通常10%的测试占用50%的运行时间）。增加并行度。将缓慢的测试移至合并后执行。检查是否有不必要的初始化/清理步骤。绘制时长随时间变化的图表，当出现阶跃变化（一周内增加>20%）时发出警报。

Defect Metrics

缺陷指标

Are we catching bugs before users do? (lagging)

我们是否在用户发现前捕获漏洞？（滞后指标）

Defect Escape Rate

缺陷逃逸率

The percentage of defects found in production relative to all defects found.

Defect escape rate = (defects found in production / total defects found) × 100

Targets: Excellent: <5%. Acceptable: 5-10%. Needs work: 10-20%. Critical: >20%.

Worked example: A release surfaces 15 total defects, 2 of them in production →

2 / 15 × 100 = 13.3%

escape rate, which lands in "needs work."

Why it matters: The single most important quality metric. It directly measures whether testing catches bugs before users do.

When it goes red: Classify escaped defects by layer (unit? integration? E2E? review?). Write a retrospective test for each. Identify if escapes cluster in specific areas — those need targeted investment.

How to track: Tag production bugs (

escaped-defect

label). Count escaped vs. pre-release defects at sprint retros.

生产环境中发现的缺陷占所有发现缺陷的比例。

Defect escape rate = (defects found in production / total defects found) × 100

目标值： 优秀：<5%。可接受：5-10%。需改进：10-20%。严重：>20%。

示例： 某次发布共发现15个缺陷，其中2个在生产环境中发现 →

2 / 15 × 100 = 13.3%

逃逸率，属于“需改进”级别。

重要性： 这是最重要的质量指标，直接衡量测试是否能在用户发现前捕获漏洞。

指标异常时的行动： 按层级（单元测试？集成测试？E2E测试？代码评审？）分类逃逸缺陷。为每个逃逸缺陷编写回顾性测试。识别逃逸缺陷是否集中在特定区域——这些区域需要针对性投入。

跟踪方式： 为生产环境漏洞添加标签（

escaped-defect

）。在迭代回顾中统计逃逸缺陷与发布前缺陷的数量。

Mean Time To Resolution (MTTR)

平均修复时间（MTTR）

MTTR = sum(resolution_time for each defect) / number of defects

Targets: P0: <4 hours. P1: <24 hours. P2: <1 sprint. P3: <2 sprints.

High MTTR often signals process bottlenecks (slow review, unclear ownership, complex deploys) rather than technical difficulty. Break into phases (triage, assign, fix, deploy) to find the bottleneck. (This is your QA defect resolution time — distinct from the DORA recovery metric below.)

MTTR = sum(resolution_time for each defect) / number of defects

目标值： P0：<4小时。P1：<24小时。P2：<1个迭代。P3：<2个迭代。

高MTTR通常表明流程瓶颈（评审缓慢、职责不清、部署复杂）而非技术难度。将修复过程分解为阶段（分类、分配、修复、部署）以找到瓶颈。（这是QA的缺陷修复时间，与下文DORA的恢复指标不同。）

Defect Density

缺陷密度

Defect density = defects found / KLOC  (or per feature shipped)

Targets: Track your own baseline; industry benchmarks are 1-10 defects per KLOC. If one module has 5x the density of others, it needs refactoring or better coverage.

Defect density = defects found / KLOC  (or per feature shipped)

目标值： 跟踪团队自身基线；行业基准是每千行代码1-10个缺陷。如果某个模块的缺陷密度是其他模块的5倍，则需要重构或提升覆盖率。

Severity Distribution

缺陷严重度分布

Healthy: P0 <5%, P1 10-15%, P2 40-50%, P3 30-40%. Visualize as a stacked bar over time. Heavy P0/P1 concentration means testing misses critical issues.

健康分布：P0 <5%，P1 10-15%，P2 40-50%，P3 30-40%。随时间绘制成堆叠柱状图。P0/P1占比过高意味着测试遗漏了关键问题。

Execution Metrics

执行指标

Is our CI pipeline fast, reliable, and cost-effective? (leading)

我们的CI流水线是否快速、可靠且成本效益高？（领先指标）

CI Pipeline Duration

CI流水线时长

Total wall-clock time by stage. Targets: Lint: <2 min. Unit: <5 min. Integration: <10 min. E2E: <15 min. Full: <30 min (= sum of stages above, reconciled with the suite-duration budget).

When it goes red: Optimize the slowest stage first. Split fast checks (every push) from slow checks (PR merge). Profile setup time vs. execution time. Parallelize sequential stages.

各阶段的总挂钟时间。目标值： Lint：<2分钟。单元测试：<5分钟。集成测试：<10分钟。E2E：<15分钟。完整流水线：<30分钟（=上述阶段时长之和，与套件执行时长预算一致）。

指标异常时的行动： 首先优化最慢的阶段。将快速检查（每次推送）与慢速检查（PR合并）分离。分析初始化时间与执行时间的占比。并行化串行阶段。

CI Cost Per Run

CI成本 Per Run

CI cost per run = (compute minutes × cost per minute) + fixed costs

50 builds/day at $0.50 each = $750/month. Optimize by caching dependencies, using spot instances, right-sizing runners, and skipping unchanged suites.

CI cost per run = (compute minutes × cost per minute) + fixed costs

50 builds/day at $0.50 each = $750/month. Optimize by caching dependencies, using spot instances, right-sizing runners, and skipping unchanged suites.

Parallelism Efficiency

并行效率

Parallelism efficiency = (total sequential time / (wall-clock time × workers)) × 100

Target: >80%. Example: 4 workers finishing in 3, 3, 3, and 12 minutes → wall-clock 12, sequential total 21, so efficiency =

21 / (12 × 4) = 44%

— well under target because three workers idled 9 minutes each. Fix by splitting by estimated duration (not file count), breaking up slow test files, and using dynamic splitting (Playwright sharding, Jest

--shard

Parallelism efficiency = (total sequential time / (wall-clock time × workers)) × 100

目标值： >80%。示例：4 workers finishing in 3, 3, 3, and 12 minutes → wall-clock 12, sequential total 21, so efficiency =

21 / (12 × 4) = 44%

--shard

Process Metrics

流程指标

Is our QA process improving over time? (mix of leading and lagging)

我们的QA流程是否在持续改进？（领先与滞后指标混合）

Automation Rate

自动化率

Automation rate = (automated test cases / total test cases) × 100

Targets: Regression: 90%+. Smoke: 100%. Exploratory: 0% (by definition). Overall: 70-85%.

Manual testing does not scale. Automation compounds — once written, a test runs thousands of times. Automate the most frequently executed scenarios first.

Automation rate = (automated test cases / total test cases) × 100

目标值： 回归测试：90%+。冒烟测试：100%。探索性测试：0%（定义使然）。整体：70-85%。

手动测试无法规模化。自动化具有复利效应——一旦编写完成，测试可以运行数千次。优先自动化最常执行的场景。

Test Creation Velocity

测试创建速度

New automated tests added per sprint (net new, excluding refactors).

Target: At least 3-5 automated tests per user story shipped. A sprint with 20 features and 0 new tests signals a growing coverage gap.

New automated tests added per sprint (net new, excluding refactors).

目标值： 每个交付的用户故事至少新增3-5个自动化测试用例。如果某个迭代交付了20个功能但新增0个测试，说明覆盖缺口在扩大。

Automation ROI

自动化ROI

Manual cost     = (manual time per cycle × cycles per year) × hourly rate
Automation cost = (write time + annual maintenance) × hourly rate
ROI             = (manual cost - automation cost) / automation cost × 100

Example: Manual regression: 8 hrs/release × 26 releases × $75/hr = $15,600/yr. Automation: 120 hrs to write + 40 hrs/yr maintenance × $75/hr = $12,000 year 1, $3,000/yr after. Year 1 ROI:

(15,600 - 12,000) / 12,000 = 30%

. Year 2 ROI:

(15,600 - 3,000) / 3,000 = 420%

. Use this to justify investment to stakeholders.

Manual cost     = (manual time per cycle × cycles per year) × hourly rate
Automation cost = (write time + annual maintenance) × hourly rate
ROI             = (manual cost - automation cost) / automation cost × 100

示例： 手动回归测试：8小时/发布 × 26次发布 × 75美元/小时 = 每年15600美元。自动化测试：编写耗时120小时 + 年度维护40小时 × 75美元/小时 = 第一年12000美元，之后每年3000美元。第一年ROI：

(15600 - 12000) / 12000 = 30%

。第二年ROI：

(15600 - 3000) / 3000 = 420%

。使用此数据向利益相关方证明自动化投入的价值。

Engineering Quality Metrics (DORA)

工程质量指标（DORA）

DORA metrics are the standard vocabulary for leadership delivery dashboards. They pair with defect escape rate — DORA tracks delivery throughput; QA metrics track delivery quality.

Metric	What it measures	Benchmark (top-15%)
Lead Time for Changes	Commit → production	< 1 day
Deployment Frequency	How often you ship	Multiple per day
Failed Deployment Recovery Time (formerly MTTR)	Time to restore service after a change-induced failure	< 1 hour
Change Failure Rate	% of deploys that cause incidents	~5% (older "high performer" bar was <15%)
Rework Rate	% of deploys that are unplanned fixes for a prior bad deploy	Low and falling

DORA 2025 formalized Rework Rate as the fifth metric and regrouped the set: the first three above are throughput (recovery time moved here because fast teams just ship the fix), Change Failure Rate + Rework Rate are instability. Reliability (availability, latency, error budget against your SLOs) is tracked alongside as a separate dimension — it is where QA/escape framing meets SRE, since error rate and availability are the user-facing tail of escaped defects. DORA 2025 also retired the named Elite/High/Medium/Low tiers in favor of percentile distributions and seven team archetypes — read the column as a percentile benchmark, not a tier you "are." The Failed Deployment Recovery Time rename (2023/2024 reports) separates change-induced failures from external outages; it is a delivery metric distinct from your QA defect MTTR above.

Source: https://dora.dev/research/. Tools that surface DORA from Git/CI data: Sleuth, Faros, LinearB, Jellyfish, Swarmia.

DORA（DevOps研究与评估）指标是管理层交付仪表盘的标准术语。它们与缺陷逃逸率搭配使用——DORA跟踪交付吞吐量；QA指标跟踪交付质量。

Metric	衡量内容	Benchmark (top-15%)
Lead Time for Changes	代码提交 → 生产部署	< 1 day
Deployment Frequency	发布频率	Multiple per day
Failed Deployment Recovery Time (formerly MTTR)	变更导致故障后的服务恢复时间	< 1 hour
Change Failure Rate	引发故障的部署占比	~5% (older "high performer" bar was <15%)
Rework Rate	为修复之前不良部署而进行的非计划部署占比	Low and falling

DORA 2025正式将Rework Rate列为第五个指标，并重新分组：前三个属于吞吐量（恢复时间移至此组，因为高效团队直接发布修复）；变更失败率+返工率属于不稳定性。可靠性（可用性、延迟、SLO下的错误预算）作为单独维度跟踪——这是QA/缺陷逃逸视角与SRE的交汇点，因为错误率和可用性是缺陷逃逸在用户端的体现。DORA 2025还取消了精英/高/中/低层级命名，改用百分位分布和七种团队类型——请将该列视为百分位基准，而非你所属的“层级”。Failed Deployment Recovery Time的重命名（2023/2024报告）将变更导致的故障与外部中断区分开；它是交付指标，与上文QA缺陷的MTTR不同。

Source: https://dora.dev/research/. 从Git/CI数据中提取DORA指标的工具：Sleuth、Faros、LinearB、Jellyfish、Swarmia.

Test Impact Analysis (TIA) as a Lever

测试影响分析（TIA）作为优化手段

TIA selects which tests to run based on which code changed (using coverage data). It trades test breadth for CI cost. Track:

% of tests skipped per PR via TIA — higher means cheaper CI; if it climbs without coverage falling, TIA is paying off.
Escaped defects from skipped tests — the safety check. If non-zero, narrow the TIA selection or expand the always-run set.

Hosted: Datadog Test Optimization (TIA), CloudBees Smart Tests, NCrunch (in-IDE). Self-built: derive from coverage data + git diff. See

coverage-analysis

TIA根据代码变更情况（使用覆盖率数据）选择要运行的测试用例。它以测试广度换取CI成本。跟踪以下指标：

% of tests skipped per PR via TIA——占比越高，CI成本越低；如果占比上升但覆盖率未下降，说明TIA发挥了作用。
Escaped defects from skipped tests——安全检查。如果数值非零，缩小TIA选择范围或扩大必运行测试集。

Hosted: Datadog Test Optimization (TIA), CloudBees Smart Tests, NCrunch (in-IDE). Self-built: derive from coverage data + git diff. See

coverage-analysis

Setting Realistic Targets

设置合理目标

Targets should match your team's maturity. Chasing enterprise metrics at a seed startup wastes effort.

Metric	Startup (seed-Series A)	Growth (Series B-C)	Enterprise (public/large)
Unit test coverage	60%	75%	85%
Branch coverage	45%	60%	75%
E2E coverage (critical paths)	Top 5 flows	Top 15 flows	All P0/P1 flows
Flakiness rate	<5%	<3%	<1%
Pass rate (7-day)	>90%	>95%	>98%
Defect escape rate	<20%	<10%	<5%
MTTR (P0)	<8 hours	<4 hours	<2 hours
CI pipeline duration	<20 min	<15 min	<10 min
Automation rate	50%	75%	90%
Metrics tracked	3-5 core	8-10 with dashboards	Full suite with alerting

Progression path:

Start with 3 metrics: code coverage, flakiness rate, defect escape rate.
Add execution metrics when CI cost or speed becomes a pain point.
Add process metrics when the team grows beyond 5 engineers.
Add the full suite when quality is a product differentiator or compliance requirement.

For the full phased rollout (week-by-week), the three dashboard layouts, and the data-source extraction table, see

references/rollout.md

and

references/dashboards.md

目标值应与团队成熟度匹配。在种子轮初创公司追求企业级指标是浪费精力。

指标	初创公司（种子轮-A轮）	成长型公司（B轮-C轮）	企业级公司（上市/大型）
单元测试覆盖率	60%	75%	85%
分支覆盖率	45%	60%	75%
E2E覆盖率（关键路径）	前5个核心流程	前15个核心流程	所有P0/P1流程
不稳定率	<5%	<3%	<1%
通过率（7天）	>90%	>95%	>98%
缺陷逃逸率	<20%	<10%	<5%
MTTR（P0）	<8小时	<4小时	<2小时
CI流水线时长	<20分钟	<15分钟	<10分钟
自动化率	50%	75%	90%
跟踪的指标数量	3-5个核心指标	8-10个带仪表盘的指标	全套指标带告警

推进路径：

从3个指标开始：代码覆盖率、不稳定率、缺陷逃逸率。
当CI成本或速度成为痛点时，添加执行指标。
当团队规模超过5名工程师时，添加流程指标。
当质量成为产品差异化因素或合规要求时，添加全套指标。

如需完整的分阶段实施计划（按周推进）、三种仪表盘布局，以及数据源提取表，请查看

references/rollout.md

和

references/dashboards.md

。

Anti-Patterns

反模式

Treating coverage as quality proof. Coverage measures execution, not verification. Pair with mutation testing for truth.
Metrics without context. "Coverage is 74%" is meaningless without target (80%), trend (up from 69%), and distribution (92% critical paths, 40% admin). Always present with target, trend, and breakdown.
Gaming metrics. Trivial tests to hit coverage numbers. Counter by pairing coverage with defect escape rate — if coverage is high but defects escape, the tests lack teeth.
Too many metrics. Tracking 25 means acting on none. Start with 3. Only add a metric when you can articulate the action it triggers.
Measuring without acting. A Grafana dashboard nobody opens. If a metric does not trigger action at least once per quarter, retire it (the metric half-life check in
```
references/rollout.md
```
).
Comparing across teams. Coverage in a payment service vs. an admin tool is not comparable. Track improvement over time, not cross-team rankings.

将覆盖率等同于质量证明：覆盖率衡量的是执行情况，而非验证情况。需搭配变异测试才能获得真实结果。
脱离上下文的指标：“覆盖率74%”本身毫无意义，需结合目标值（80%）、趋势（从69%上升）和分布（关键路径92%，后台管理模块40%）。始终与目标值、趋势和细分数据一起展示。
指标造假：编写无意义的测试以达到覆盖率数值。解决方法：将覆盖率与缺陷逃逸率搭配——如果覆盖率高但缺陷仍逃逸，说明测试缺乏有效性。
指标过多：跟踪25个指标意味着无法对任何指标采取行动。从3个开始。只有当你能明确说明指标触发的行动时，才添加新指标。
只衡量不行动：无人查看的Grafana仪表盘。如果一个指标每季度至少触发一次行动，就淘汰它（
```
references/rollout.md
```
中的指标半衰期检查）。
跨团队比较：支付服务的覆盖率与后台管理工具的覆盖率不具可比性。跟踪自身随时间的改进，而非跨团队排名。

Verification

验证

The whole point is a working feedback loop — prove the data actually flows before declaring done:

Coverage emits machine-readable output. Run your coverage tool and confirm it writes JSON/LCOV (
```
coverage/coverage-final.json
```
,
```
lcov.info
```
) — not just an HTML report a human reads.
CI history is queryable. Hit your CI provider's API for the last 7 days of run results and confirm you can compute pass rate and flakiness from it.
Escape labels exist. Query the issue tracker for the
```
escaped-defect
```
(or equivalent) label and confirm at least the tagging convention is in place.
A gate actually blocks. Open a throwaway PR that drops coverage or adds a
```
.skip
```
and confirm CI fails it — a gate that never fires proves nothing.

核心目标是建立有效的反馈闭环——在宣布完成前，需证明数据确实在流转：

覆盖率输出机器可读格式：运行覆盖率工具，确认它生成JSON/LCOV格式文件（
```
coverage/coverage-final.json
```
、
```
lcov.info
```
）——而非仅生成人类可读的HTML报告。
CI历史可查询：调用CI提供商的API获取过去7天的运行结果，确认你可以从中计算通过率和不稳定率。
逃逸标签已存在：查询缺陷跟踪工具中的
```
escaped-defect
```
（或等效）标签，确认至少标签规范已到位。
门禁实际生效：创建一个临时PR，降低覆盖率或添加
```
.skip
```
，确认CI失败——从未触发的门禁毫无意义。

Done When

完成标准

Key metrics are defined with explicit formulas: coverage % (line and branch), flakiness rate, defect escape rate, and MTTR by severity.
Baseline values are established for each metric from at least 2 weeks of collected data, with targets set per the company-stage table.
Data collection is automated via CI integrations (coverage from test runner, flakiness from CI run history, defects from issue-tracker labels) — no manual steps.
Quality gates are configured in CI to block merges or deployments when flakiness exceeds threshold, coverage drops, or critical tests fail (verified by a failing throwaway PR).
Each metric has a named owner, and a metrics block exists in the recurring retro template (or a standing calendar invite) — a checkable artifact, not just an intention.

关键指标已明确定义并配有计算公式：覆盖率（行和分支）、不稳定率、缺陷逃逸率，以及按严重度划分的MTTR。
每个指标的基线值已基于至少2周的收集数据确定，并根据公司阶段表设置了目标值。
数据收集通过CI集成自动化完成（覆盖率来自测试运行器，不稳定率来自CI运行历史，缺陷来自缺陷跟踪工具标签）——无手动步骤。
CI中已配置质量门禁，当不稳定率超过阈值、覆盖率下降或关键测试失败时阻止合并或部署（通过临时失败PR验证）。
每个指标都有指定负责人，且迭代回顾模板（或固定日历会议）中包含指标模块——这是可检查的工件，而非仅为意向。

Reference Files (in

references/

)

参考文件（位于

references/

目录）

dashboards.md — the engineering (daily), leadership (monthly), and sprint-health dashboard layouts, plus the release-confidence-score example weighting.
rollout.md — the four-phase week-by-week implementation plan, the data-source extraction table, and the metric half-life retirement practice.

dashboards.md——面向工程师（日常）、管理层（月度）和迭代健康度的仪表盘布局，以及发布信心评分的示例权重。
rollout.md——四阶段按周实施计划、数据源提取表，以及指标半衰期淘汰实践。