release-readiness

Compare original and translation side by side

🇺🇸

Original

English

🇨🇳

Translation

Chinese

<objective> "I think it's fine" ships the bug that the rollback you never practiced can't undo at 6 PM on a Friday. This skill turns "ready to ship" into something measurable: a go/no-go checklist with evidence for each item, a sub-5-minute smoke suite, a staged rollout with metric-gated promotion, rollback thresholds defined before deploy, and post-deployment verification. Every section gives concrete criteria, not aspirations. </objective>

<objective> “我觉得没问题”这种判断会导致你在周五下午6点发布带有bug的版本，而你从未演练过的回滚根本无法撤销。本技能将“准备发布”转化为可衡量的指标：一份每项都有实证支持的上线/不上线检查清单、耗时不到5分钟的冒烟测试套件、基于指标管控的分阶段发布、部署前就定义好的回滚阈值，以及部署后验证。每个部分都提供具体标准，而非空泛的目标。 </objective>

Discovery Questions

探索问题

Check

.agents/qa-project-context.md

first — if it exists, use it and skip anything answered there. Then ask only what's missing.

Release cadence and process:

How often do you release, and who makes the go/no-go call? (Continuous/daily/weekly vs engineering lead/QA lead/release manager — sets how heavyweight the checklist should be.)
Is there a release train schedule or is it ad-hoc?
How many environments exist between dev and production? (staging, pre-prod, canary)

Current state:

What does the current go/no-go process look like, and is it documented?
Has a release ever been rolled back? How long did it take? (Reveals whether rollback is real or aspirational.)
What was the last release incident and its root cause? Any release-blocking bugs right now?

Infrastructure and capabilities:

Do you have rollback capability, and how long does a rollback take?
Can you do staged/canary deployments?
Do you have feature flags, and how are they managed? (Decides flag-based vs infra-level rollout.)
What monitoring and alerting is in place? Are database migrations reversible?

Team and communication:

Who is on-call during and after releases?
How are stakeholders notified, and is there a release communication channel?
How are release notes generated?

先查看

.agents/qa-project-context.md

——如果存在，使用其中的内容并跳过已回答的问题。然后仅询问缺失的信息。

发布节奏与流程：

你们多久发布一次，由谁做出上线/不上线的决策？（持续发布/每日/每周 vs 工程负责人/QA负责人/发布经理——决定检查清单的复杂程度。）
是否有固定的发布列车时间表，还是按需发布？
从开发环境到生产环境之间有多少个环境？（预发布环境、准生产环境、金丝雀环境）

当前状态：

当前的上线/不上线流程是什么样的，是否有文档记录？
有没有过发布后回滚的情况？回滚耗时多久？（揭示回滚是实际可行的还是仅停留在设想层面。）
最近一次发布事故是什么，根本原因是什么？当前有没有阻碍发布的bug？

基础设施与能力：

你们具备回滚能力吗，回滚耗时多久？
是否可以进行分阶段/金丝雀部署？
是否使用功能开关（feature flag），如何管理？（决定基于开关还是基础设施层面的发布方式。）
有哪些监控和告警机制？数据库迁移是否可逆？

团队与沟通：

发布期间及发布后谁负责值班？
如何通知相关利益方，是否有专门的发布沟通渠道？
发布说明如何生成？

Core Principles

核心原则

1. Release confidence comes from evidence, not feelings

1. 发布信心来自实证，而非主观感受

"I think it's fine" is not a go/no-go criterion. Evidence means: all CI pipelines green, smoke tests pass on staging, performance budgets met, no open P0/P1 bugs. If you can't point to data, you're not ready.

“我觉得没问题”不是上线/不上线的判断标准。实证指：所有CI流水线执行成功、预发布环境冒烟测试通过、性能符合预算、无未解决的P0/P1级bug。如果无法提供数据支撑，说明尚未准备就绪。

2. Smoke tests are the last safety net, not the only safety net

2. 冒烟测试是最后一道安全防线，而非唯一防线

Smoke tests catch catastrophic failures. They are not a substitute for thorough testing throughout the development cycle. If your smoke test suite is the only thing between you and production, you have a process problem upstream.

冒烟测试用于发现灾难性故障，不能替代开发周期中的全面测试。如果冒烟测试套件是你与生产环境之间的唯一屏障，说明上游流程存在问题。

3. Staged rollouts reduce blast radius

3. 分阶段发布缩小影响范围

Deploying to 100% of users simultaneously means 100% of users are affected by any bug. Staged rollouts (canary, percentage-based, ring-based) let you catch issues when they affect 1% of users instead of all of them.

一次性部署给100%用户意味着任何bug都会影响所有用户。分阶段发布（金丝雀、按百分比、按圈层）可以让你在bug仅影响1%用户时就发现问题，而非波及全部用户。

4. Rollback criteria must be defined BEFORE release

4. 回滚标准必须在发布前定义

If you wait until something is on fire to decide whether to roll back, you'll waste critical minutes debating. Define the criteria in advance, relative to baseline: "If error rate exceeds 2x baseline within 15 minutes, we roll back. No discussion needed." Tie the trigger to your DORA targets — a release whose error rate would push you past your change-failure-rate target, or whose recovery would blow your MTTR target, is one the rollback rule exists to stop.

如果等到问题爆发才决定是否回滚，你会浪费关键时间进行争论。提前定义相对于基准线的标准：“如果15分钟内错误率超过基准线的2倍，立即回滚，无需讨论。”将触发条件与你的DORA指标绑定——如果某个发布的错误率会让你超出变更失败率目标，或者恢复时间会突破平均恢复时间（MTTR）目标，那么回滚规则就是为阻止这类发布而存在的。

5. Every release is a learning opportunity

5. 每一次发布都是学习机会

Post-deployment verification isn't just about catching bugs. Track what went well, what was slow, what was stressful. Improve the process continuously.

部署后验证不仅是为了发现bug，还要记录哪些环节进展顺利、哪些环节耗时较长、哪些环节压力较大。持续改进流程。

Go/No-Go Checklist

上线/不上线检查清单

Use this as a template. Adapt it to your context. Every item should be verifiable with evidence, not just "I checked." Store the completed checklist as a versioned artifact (e.g.

RELEASE-<version>.md

or a tracked issue) so sign-off is auditable.

将此作为模板，根据你的场景调整。每一项都必须可通过实证验证，而非仅“我已检查”。将填写完成的清单作为版本化工件存储（例如

RELEASE-<version>.md

或跟踪工单），以便签核可审计。

Automated Checks (Must Pass)

自动化检查（必须全部通过）

All CI pipelines green — Unit tests, integration tests, E2E tests, type checking, linting
Smoke test suite passes on staging — Critical user journeys verified in the staging environment
No open P0/P1 bugs for this release — Check issue tracker, filter by milestone/label
Performance budgets met — API response times and bundle size within thresholds; Lighthouse CI for frontend releases (skip Lighthouse for API/backend-only releases — it measures page load, not service health)
Security scan clean — No high/critical vulnerabilities in
```
npm audit
```
/ Snyk / Dependabot
API contract tests pass — No breaking changes to public APIs
Visual regression tests pass — No unintended visual changes
Accessibility checks pass — axe-core scan shows no new violations

所有CI流水线执行成功 —— 单元测试、集成测试、端到端测试、类型检查、代码扫描
预发布环境冒烟测试套件通过 —— 验证预发布环境中的关键用户旅程
本次发布无未解决的P0/P1级bug —— 检查问题跟踪工具，按里程碑/标签筛选
性能符合预算 —— API响应时间和包大小在阈值范围内；前端发布需通过Lighthouse CI（纯API/后端发布可跳过Lighthouse——它衡量页面加载速度，而非服务健康状况）
安全扫描无风险 ——
```
npm audit
```
/Snyk/Dependabot未发现高/严重级漏洞
API契约测试通过 —— 公共API无破坏性变更
视觉回归测试通过 —— 无意外的视觉变更
无障碍检查通过 —— axe-core扫描未发现新的违规项

Manual Checks (Verify Before Go)

手动检查（上线前验证）

Risk Assessment

风险评估

Change scope categorized — Small (config change, copy update), Medium (new feature, refactor), Large (architecture change, migration)
Blast radius estimated — What percentage of users could be affected if something goes wrong?
Revert complexity assessed — Can this be reverted in <5 minutes? Does reverting require a data migration?

变更范围已分类 —— 小型（配置变更、文案更新）、中型（新功能、重构）、大型（架构变更、迁移）
影响范围已估算 —— 如果出现问题，会影响多大比例的用户？
回滚复杂度已评估 —— 是否能在5分钟内完成回滚？回滚是否需要数据迁移？

Smoke Test Suite Design

冒烟测试套件设计

What to Include

需包含的内容

Smoke tests cover critical user journeys only. If these fail, the application is fundamentally broken.

Typical smoke test suite (5-8 tests):

Application health — Homepage loads, returns 200, no JavaScript errors in console
Authentication — User can log in with valid credentials, session is established
Core workflow — The primary value-delivering action works (e.g., create a document, submit a form, add to cart or complete a purchase flow)
Data retrieval — Key data loads correctly (dashboard populates, search returns results, product page loads)
Payment/transaction (if applicable) — Payment flow completes with test credentials
API health — Primary API endpoints return valid responses with correct schemas
Navigation — Critical navigation paths work (deep links, redirects, menu items)
Error handling — Application shows a user-friendly error page for invalid routes (404)

冒烟测试仅覆盖关键用户旅程。如果这些测试失败，说明应用存在根本性故障。

典型冒烟测试套件（5-8个测试用例）：

应用健康状态 —— 首页加载成功，返回200状态码，控制台无JavaScript错误
身份验证 —— 用户可使用有效凭据登录，会话已建立
核心工作流 —— 核心价值交付动作正常（例如创建文档、提交表单、添加到购物车或完成购买流程）
数据检索 —— 关键数据加载正常（仪表板填充、搜索返回结果、产品页面加载）
支付/交易（如有）—— 使用测试凭据完成支付流程
API健康状态 —— 核心API端点返回有效响应，且符合正确的 schema
导航 —— 关键导航路径正常（深层链接、重定向、菜单项）
错误处理 —— 应用对无效路由显示友好的错误页面（404）

What NOT to Include

无需包含的内容

Edge cases (those belong in regression tests)
Visual perfection (that belongs in visual regression tests)
Performance benchmarks (that belongs in performance tests)
Exhaustive form validation (that belongs in unit/integration tests)

边缘场景（属于回归测试范畴）
视觉完美度（属于视觉回归测试范畴）
性能基准（属于性能测试范畴）
详尽的表单验证（属于单元/集成测试范畴）

Keeping It Fast

确保测试速度

Target: under 5 minutes for the entire smoke suite.

Run tests in parallel where possible
Use API calls instead of UI interactions for setup (create test user via API, not through registration form)
Skip non-critical assertions (don't check exact copy text, check that elements exist)
Use a dedicated test account with pre-created data (don't create data from scratch each run)
Avoid unnecessary waits — use smart waiting (wait for element, not
```
sleep(3000)
```
/
```
waitForTimeout
```
)

目标：整个冒烟测试套件耗时不超过5分钟。

尽可能并行运行测试
使用API调用而非UI交互完成前置操作（通过API创建测试用户，而非注册表单）
跳过非关键断言（不检查精确文案，只需确认元素存在）
使用带有预创建数据的专用测试账号（无需每次从头创建数据）
避免不必要的等待——使用智能等待（等待元素加载，而非
```
sleep(3000)
```
/
```
waitForTimeout
```
）

Environment-Specific Smoke Tests

环境专属冒烟测试

Staging smoke tests:

Full smoke suite (all 5-8 tests)
Can use test payment providers
Can test with feature flags in upcoming release configuration
Can test database migrations

Production smoke tests:

Subset of staging smoke tests (3-5 tests)
Use synthetic test accounts (clearly labeled, won't affect analytics)
Never test with real payment transactions (use sandbox mode or skip)
Focus on: app loads, auth works, core read operations work, API responds

Post-deployment smoke tests:

Run immediately after deploy completes (within 60 seconds)
Same as production smoke tests
If any fail, trigger alert and begin rollback evaluation

Staging is not production: it has different data volumes, traffic patterns, third-party configurations, and infrastructure scale. That gap is exactly why production and post-deployment smoke tests exist on top of staging verification.

预发布环境冒烟测试：

完整冒烟测试套件（全部5-8个测试用例）
可使用测试支付服务商
可测试即将发布版本的功能开关配置
可测试数据库迁移

生产环境冒烟测试：

预发布环境冒烟测试的子集（3-5个测试用例）
使用合成测试账号（明确标记，不会影响分析数据）
绝不测试真实支付交易（使用沙箱模式或跳过）
重点关注：应用加载、身份验证、核心读取操作、API响应

部署后冒烟测试：

部署完成后立即运行（60秒内）
与生产环境冒烟测试内容一致
如果任何测试失败，触发告警并开始评估回滚

预发布环境不等于生产环境——两者的数据量、流量模式、第三方配置和基础设施规模都不同。这种差异正是为什么在预发布验证之外，还需要生产环境和部署后冒烟测试的原因。

Staged Rollout Validation

分阶段发布验证

Rollout Stages

发布阶段

A typical staged rollout. The same ladder expressed for flag-based rollouts adds a 25% step (see below):

Stage	Traffic %	Duration	Purpose
Canary	1%	15-30 min	Catch crashes, exceptions, obvious failures
Early adopters	10%	1-2 hours	Validate error rates, latency, business metrics
Partial rollout	25-50%	2-4 hours	Confirm stability at scale
Full rollout	100%	—	Monitor for 24 hours post-deployment

典型的分阶段发布流程。基于功能开关的发布流程在此基础上增加25%的步骤（见下文）：

阶段	流量占比	持续时间	目的
金丝雀	1%	15-30分钟	捕获崩溃、异常、明显故障
早期尝鲜用户	10%	1-2小时	验证错误率、延迟、业务指标
部分发布	25-50%	2-4小时	确认大规模部署下的稳定性
全面发布	100%	—	部署后监控24小时

What to Monitor Between Stages

阶段间需监控的内容

Before promoting to the next stage, verify all of these:

Error metrics:

Error rate (HTTP 5xx) is not higher than baseline
Exception count is not higher than baseline
No new error types appearing in logs

Performance metrics:

P50 and P95 latency are within acceptable range (relative to baseline, not an absolute ceiling)
No increase in timeout errors
Database query times are stable

Business metrics:

Conversion rate is not dropping
User engagement (page views, actions) is stable
Revenue/transaction volume is normal (if applicable)

Infrastructure metrics:

CPU and memory usage are normal
No increase in queue depth or message backlog
No disk space issues from new logging

推进到下一阶段前，需验证所有以下指标：

错误指标：

错误率（HTTP 5xx）不高于基准线
异常数量不高于基准线
日志中未出现新的错误类型

性能指标：

P50和P95延迟在可接受范围内（相对于基准线，而非绝对上限）
超时错误未增加
数据库查询时间稳定

业务指标：

转化率未下降
用户参与度（页面浏览量、操作次数）稳定
收入/交易量正常（如有）

基础设施指标：

CPU和内存使用正常
队列深度或消息积压未增加
新日志未导致磁盘空间问题

Automated Promotion Criteria

自动推进标准

Define rules for automatic promotion between stages. Each gate combines an error-rate ceiling, a latency ceiling expressed relative to baseline, a stability window, and (at higher stages) business-metric guardrails. See

references/rollout-automation.md

for the full canary→10%→50%→100% promotion ruleset.

定义阶段间自动推进的规则。每个关卡结合错误率上限、相对于基准线的延迟上限、稳定窗口，以及（在更高阶段）业务指标防护规则。完整的金丝雀→10%→50%→100%推进规则请参考

references/rollout-automation.md

。

Feature Flag Gradual Rollout

基于功能开关的渐进式发布

An alternative to infrastructure-level canary deploys:

Deploy new code to 100% with the feature flag OFF
Enable the flag for internal users first (dogfooding)
Enable for 1% of users (canary equivalent)
Gradually increase: 10%, 25%, 50%, 100%
Remove the flag after full rollout is stable for 1 week

Advantages: Faster rollback (just flip the flag), no infrastructure changes, can target specific user segments.

Disadvantages: Code complexity (branching logic), stale flags become tech debt, doesn't catch infrastructure issues.

这是基础设施层面金丝雀部署的替代方案：

将新代码部署到100%服务器，但功能开关处于关闭状态
首先为内部用户启用开关（内部测试）
为1%的用户启用开关（等效于金丝雀阶段）
逐步扩大范围：10%、25%、50%、100%
全面发布稳定运行1周后移除开关

优势： 回滚更快（只需切换开关），无需变更基础设施，可针对特定用户群体

劣势： 代码复杂度增加（分支逻辑），过期开关会变成技术债务，无法捕获基础设施问题

Tooling

工具选型

Platform	Best at	Notes
LaunchDarkly	Enterprise scale; Guarded Rollouts (auto-canary analysis, GA since May 2025); AI Configs for prompt/model rollouts; agent graphs	Acquired Highlight.io in 2025 — also offers observability tied to flags
Statsig	Experiment-first culture; Switchback experiments (Feb 2026 update) for two-sided marketplaces; auto-tune	Acquired by OpenAI Sept 2025; still operates independently as of mid-2026, but weigh acquisition/roadmap risk before a multi-year infrastructure bet
GrowthBook	OSS-first; stale-flag detection with code-reference scanning; SQL-based experimentation	Strong fit when you want to self-host and avoid vendor lock-in
Unleash	OSS, GitOps-style flag definition, environment scoping	Apache 2 license; Enterprise tier for SSO/audit
Flagsmith	Kill switches as first-class concept; canary alerts; OSS option	Published "what is a kill switch" + "release testing" guides 2026
Harness FME (formerly Split)	Targeted rollouts + monitoring tied to deploy pipelines; warehouse-native experimentation + flag archiving (2026)	Rebranded after Harness acquisition

Vendor-native canary analysis (LaunchDarkly Guarded Rollouts, Statsig Auto-tune, Flagger) is now common — if your platform offers it, prefer it over hand-rolled rollout-policy YAML. Given the vendor churn this table documents (Statsig→OpenAI, Split→Harness FME), prefer OpenFeature-compatible SDKs (CNCF spec; Harness FME and others now standardize on it) so flag tooling stays swappable.

平台	擅长场景	说明
LaunchDarkly	企业级规模；Guarded Rollouts（自动金丝雀分析，2025年5月正式发布）；AI Configs用于提示词/模型发布；Agent图谱	2025年收购Highlight.io——还提供与开关绑定的可观测性功能
Statsig	以实验为核心的文化；Switchback experiments（2026年2月更新）适用于双边市场；自动调优	2025年9月被OpenAI收购；截至2026年中期仍独立运营，但在进行多年基础设施投入前需考量收购/路线图风险
GrowthBook	开源优先；通过代码引用扫描检测过期开关；基于SQL的实验	适合希望自托管、避免厂商锁定的场景
Unleash	开源、GitOps风格的开关定义、环境作用域	Apache 2许可证；企业版提供SSO/审计功能
Flagsmith	将终止开关作为核心概念；金丝雀告警；开源选项	2026年发布了“什么是终止开关”+“发布测试”指南
Harness FME（原Split）	定向发布+与部署流水线绑定的监控；基于数据仓库的实验+开关归档（2026年）	被Harness收购后更名

厂商原生的金丝雀分析（LaunchDarkly Guarded Rollouts、Statsig Auto-tune、Flagger）现在已很普遍——如果你的平台提供该功能，优先使用而非自行编写发布策略YAML。考虑到本表记录的厂商变动（Statsig→OpenAI、Split→Harness FME），优先选择兼容OpenFeature的SDK（CNCF规范；Harness FME等厂商现已标准化支持），以便开关工具可灵活替换。

Rolling Out AI/LLM Features

AI/LLM功能发布

AI features need a distinct rollout pattern: prompt versions and model IDs are configurable separately from code, and a kill switch is mandatory.

Pin the prompt template version and model ID in your AI Configs platform (LaunchDarkly AI Configs, custom dataset, or feature-flag JSON).
Roll out the prompt/model combo behind a flag — internal first, then 1%, 10%, etc.
Watch eval metrics (hallucination rate, jailbreak success rate, cost per request) per cohort, not just error rate.
Cost guardrail: a budget circuit breaker that fails the feature open (graceful fallback) when a model's per-request cost spikes.
Kill switch: a single flag that disables the AI path and routes to a deterministic fallback or a "feature unavailable" state — testable in staging before launch.

See

ai-system-testing

for prompt-level eval test patterns and

testing-in-production

for canary metric design.

AI功能需要独特的发布模式：提示词版本和模型ID可独立于代码配置，且终止开关是必需的。

在AI配置平台（LaunchDarkly AI Configs、自定义数据集或功能开关JSON）中固定提示词模板版本和模型ID。
将提示词/模型组合通过开关发布——首先面向内部用户，然后是1%、10%等比例用户。
按群组监控评估指标（幻觉率、越狱成功率、每请求成本），而非仅错误率。
成本防护：设置预算断路器，当模型每请求成本飙升时，自动降级到优雅降级方案（返回确定性 fallback 或“功能不可用”状态）。
终止开关：一个单独的开关，可禁用AI路径并路由到确定性 fallback 或“功能不可用”状态——发布前需在预发布环境测试。

提示词级评估测试模式请参考

ai-system-testing

，金丝雀指标设计请参考

testing-in-production

。

Rollback Criteria and Process

回滚标准与流程

Automated Rollback Triggers

自动回滚触发条件

Define these thresholds BEFORE deployment. When any trigger fires, rollback begins automatically. All thresholds are relative to the measured baseline, not absolute ceilings.

Metric	Threshold	Action
Error rate (5xx)	>2x baseline for 5 min	Auto-rollback
P95 latency	>3x baseline for 5 min	Auto-rollback
Health check	3 consecutive failures	Auto-rollback
Crash rate (mobile)	>0.5%	Auto-rollback
Error budget	>50% burned in 1 hour	Auto-rollback

部署前定义以下阈值。当任何触发条件满足时，自动开始回滚。所有阈值均相对于实测基准线，而非绝对上限。

指标	阈值	动作
错误率（5xx）	5分钟内超过基准线2倍	自动回滚
P95延迟	5分钟内超过基准线3倍	自动回滚
健康检查	连续3次失败	自动回滚
崩溃率（移动端）	>0.5%	自动回滚
错误预算	1小时内消耗超过50%	自动回滚

Manual Rollback Triggers

手动回滚触发条件

These require human judgment but should have clear guidelines:

Customer-reported critical issue — Multiple users reporting the same problem
Data integrity concern — Evidence of corrupted or incorrect data
Security vulnerability discovered — Active exploitation or high-severity CVE
Monitoring blind spots — You realize you can't monitor a critical metric for the new feature
On-call engineer judgment — The on-call engineer always has authority to trigger a rollback

这些需要人工判断，但应有明确的指导原则：

用户反馈的严重问题 —— 多名用户报告相同问题
数据完整性担忧 —— 存在数据损坏或错误的证据
发现安全漏洞 —— 存在主动利用或高严重性CVE
监控盲区 —— 发现无法监控新功能的关键指标
值班工程师判断 —— 值班工程师始终有权触发回滚

Rollback Procedure

回滚流程

Step 1: Decide (< 2 minutes)

Is the trigger automated or manual?
If manual: does the issue meet rollback criteria? If yes, proceed. Don't debate.

Step 2: Execute rollback (< 5 minutes)

Kill switch (fastest, prefer if available): Flip the dedicated kill-switch flag for the affected feature. Distinct from a full code rollback — disables one capability without redeploy. Test the kill switch in staging before every release; an untested switch is not a switch.
Feature flag rollback: Disable the feature flag for the new code path. Slower than a kill switch when both exist (rollout flag vs kill switch are different concerns).
Code rollback: Revert to the previous deployment (re-deploy previous image/artifact). Use when the issue is not contained to a single flagged feature.
Database rollback: Run backward migration if applicable. If migration is irreversible, skip this step and handle data separately.
Cache invalidation: Clear CDN and application caches if the old version would serve stale/incorrect data.

Step 3: Verify (< 5 minutes)

Run production smoke tests
Verify error rate returns to baseline
Check that the rolled-back version serves correctly

Step 4: Communicate (< 10 minutes)

Notify the release channel using the rollback skeleton below
Update status page if user-facing impact occurred
Brief the support team

Step 5: Investigate (next business day)

Root cause analysis
Write a regression test that would have caught the issue
Update the go/no-go checklist if a check was missing
Schedule the fix and re-release

步骤1：决策（<2分钟）

触发条件是自动还是手动？
如果是手动：问题是否符合回滚标准？如果是，立即执行，无需争论。

步骤2：执行回滚（<5分钟）

终止开关（最快，优先使用如果可用）： 切换受影响功能的专用终止开关。与完整代码回滚不同——无需重新部署即可禁用单个功能。每次发布前需在预发布环境测试终止开关；未测试的开关形同虚设。
功能开关回滚： 禁用新代码路径的功能开关。当同时存在终止开关时，此方式比终止开关慢（发布开关与终止开关是不同的关注点）。
代码回滚： 回滚到上一个部署版本（重新部署上一个镜像/工件）。适用于问题不限于单个开关控制的功能的情况。
数据库回滚： 如有必要，执行反向迁移。如果迁移不可逆，跳过此步骤，单独处理数据问题。
缓存失效： 如果旧版本会提供过期/错误数据，清除CDN和应用缓存。

步骤3：验证（<5分钟）

运行生产环境冒烟测试
验证错误率回到基准线
确认回滚后的版本正常服务

步骤4：沟通（<10分钟）

使用以下回滚模板在发布频道通知
如果影响用户，更新状态页面
向支持团队通报情况

步骤5：调查（下一个工作日）

根本原因分析
编写可捕获该问题的回归测试
如果缺失检查项，更新上线/不上线检查清单
安排修复和重新发布

Rollback announcement skeleton

回滚通知模板

Drop this in the release channel during Step 4. For the full release + rollback templates, see

references/communication-templates.md

Subject: [Rollback] v{version} — {date} {time}
Status: ROLLED BACK
Reason: {one line — e.g. error rate 4x baseline within 8 min}
Impact: {who was affected, for how long}
Current state: Running previous version v{prev_version}
Next steps:
- Root cause investigation: {owner}
- Fix ETA: {estimate or "investigating"}

步骤4中可将以下内容发布到发布频道。完整的发布+回滚模板请参考

references/communication-templates.md

。

主题：[回滚] v{version} — {日期} {时间}
状态：已回滚
原因：{一句话说明——例如8分钟内错误率达基准线4倍}
影响：{受影响人群，持续时间}
当前状态：运行上一版本v{prev_version}
下一步：
- 根本原因调查：{负责人}
- 修复预计时间：{预估时间或“调查中”}

Data Considerations

数据考量

When a migration can't be rolled back:

Forward-fix: Deploy a fix on top of the current (broken) version instead of rolling back
Dual-write: During migration, write to both old and new schemas; rollback drops the new writes
Shadow migration: Migrate in the background, validate, then cut over. Rollback just stops the cutover
Point-in-time recovery: Restore database from backup (last resort, causes data loss for changes since backup)

当迁移无法回滚时：

正向修复： 在当前（损坏）版本上部署修复，而非回滚
双写： 迁移期间同时写入旧和新schema；回滚时丢弃新写入的数据
影子迁移： 在后台进行迁移，验证后再切换；回滚只需停止切换
时间点恢复： 从备份恢复数据库（最后手段，会丢失备份后的数据变更）

Post-Deployment Verification

部署后验证

Immediate (0-15 minutes)

即时验证（0-15分钟）

Short-term (15 minutes - 2 hours)

短期验证（15分钟-2小时）

Medium-term (2-24 hours)

中期验证（2-24小时）

Anti-Patterns

反模式

"It worked on staging"

“预发布环境没问题”

Staging is not production — different data volumes, traffic patterns, third-party configurations, and infrastructure scale. Staging success is necessary but not sufficient evidence of readiness. Fix: Use production smoke tests and staged rollouts in addition to staging verification.

预发布环境不等于生产环境——两者的数据量、流量模式、第三方配置和基础设施规模都不同。预发布环境验证通过是必要条件，但不足以证明发布就绪。 修复方案： 在预发布验证之外，使用生产环境冒烟测试和分阶段发布。

No rollback plan

无回滚计划

"We'll figure it out if something goes wrong" means you'll figure it out under pressure, sleep-deprived, with users complaining. That's when mistakes happen. Fix: Document the rollback procedure. Practice it quarterly. Time it. Make it a checklist, not tribal knowledge.

“出问题时再想办法”意味着你会在压力下、睡眠不足、用户抱怨的情况下解决问题，这时很容易出错。 修复方案： 文档化回滚流程，每季度演练一次，记录耗时。将其做成检查清单，而非依赖团队经验。

Deploying on Friday afternoon

周五下午部署

You deploy at 4 PM on Friday. An issue surfaces at 6 PM. Your team is at dinner. The issue grows overnight. Monday morning is chaos. Fix: Deploy early in the week, early in the day, when the full team is available to monitor. If you must deploy on Friday, deploy before noon with extra monitoring.

你在周五下午4点部署，6点出现问题，团队正在吃晚餐，问题一夜之间恶化，周一早上一片混乱。 修复方案： 在一周的早间、一天的早间部署，此时整个团队都可参与监控。如果必须在周五部署，需在中午前完成并增加监控。

Skipping smoke tests because "the pipeline is green"

因为“流水线执行成功”而跳过冒烟测试

CI pipelines test against test data in test environments. Smoke tests verify the deployed application works with production configuration, production data, and production infrastructure. Fix: Smoke tests are non-negotiable. If they're slow, make them faster. If they're flaky, fix them. Never skip them.

CI流水线在测试环境针对测试数据进行测试，冒烟测试验证部署后的应用在生产配置、生产数据和生产基础设施下是否正常工作。 修复方案： 冒烟测试是必不可少的。如果测试速度慢，优化速度；如果测试不稳定，修复问题。绝不跳过。

Big-bang releases instead of incremental

大爆炸式发布而非增量发布

Accumulating 6 weeks of changes into one mega-release means: more things can break, harder to identify which change caused the issue, higher risk, longer rollback time, more stress. Fix: Release smaller, more frequently. If you can't do continuous deployment, aim for weekly or bi-weekly releases with small, well-understood changesets.

将6周的变更累积成一个大型发布意味着：更多可能出错的地方、更难定位问题根源、更高风险、更长回滚时间、更大压力。 修复方案： 发布更小、更频繁的版本。如果无法进行持续部署，目标是每周或每两周发布一次，变更集小且易于理解。

No post-deployment verification

无部署后验证

You deploy and move on to the next feature. An hour later, users are experiencing errors that nobody is watching for. Fix: Assign someone to monitor dashboards for 30-60 minutes post-deploy. Set up automated alerts with appropriate thresholds. Run post-deployment smoke tests.

部署后就转向下一个功能，一小时后用户遇到错误，但无人监控。 修复方案： 指定人员在部署后监控仪表盘30-60分钟。设置带有适当阈值的自动告警。运行部署后冒烟测试。

Rollback aversion

不愿回滚

"We're so close to fixing it, let's just push a hotfix forward." Meanwhile, users are affected for another 45 minutes while you debug under pressure. Fix: Roll back first, investigate second. A working previous version is better than a broken current version. Your ego can recover; user trust is harder to rebuild.

“我们快修复好了，直接推送热修复吧。”与此同时，用户在你压力下调试的45分钟内持续受影响。 修复方案： 先回滚，再调查。可用的旧版本比损坏的当前版本更好。你的自尊心可以恢复，但用户信任很难重建。

Feature flag accumulation

功能开关堆积

You use feature flags for safe rollouts (good!) but never remove them (bad). After a year, you have 200 flags, nobody knows which are active, and flag interactions cause mysterious bugs. Fix: 2026 best practice is platform-level stale detection, not calendar reminders. Use GrowthBook stale-flag detection (code references), LaunchDarkly archive flow, or Flagsmith's flag age telemetry to surface flags whose code paths haven't been touched in N weeks. Pair with a quarterly review where flags older than the threshold are either archived or get a documented owner + reason to keep. Calendar dates rot; code-reference scans don't.

你使用功能开关进行安全发布（很好！）但从不移除（很糟！）。一年后，你有200个开关，没人知道哪些是启用的，开关交互导致神秘bug。 修复方案： 2026年的最佳实践是平台级过期检测，而非日历提醒。使用GrowthBook的过期开关检测（代码引用）、LaunchDarkly的归档流程或Flagsmith的开关使用时长遥测，找出代码路径已N周未触碰的开关。配合季度评审，超过阈值的开关要么归档，要么指定文档化的负责人+保留理由。日历日期会失效，但代码引用扫描不会。

Canary alerts that lie

金丝雀告警不准确

Auto-rollback wired to a metric that's noisy, late-arriving, or partially aggregated. The alert fires (or doesn't) at the wrong time, and the team learns to mistrust it — so when the real incident arrives, the signal is ignored. Fix: Treat the canary alert like any other test — it has a false-positive rate and a false-negative rate, and you measure both. Run a "shadow" period where the alert publishes its decision but doesn't actually rollback; compare its calls to ground truth for two weeks. Promote to auto-rollback only after the false-positive rate is below your tolerance. Reference: https://www.flagsmith.com/blog/when-canary-alerts-go-wrong

自动回滚绑定的指标存在噪声、延迟或部分聚合。告警在错误时间触发（或不触发），团队开始不信任它——因此当真正的事故发生时，信号被忽略。 修复方案： 将金丝雀告警视为测试——它有假阳性率和假阴性率，你需要衡量两者。运行“影子”阶段，告警仅发布决策但不实际回滚；将其决策与真实情况对比两周。只有当假阳性率低于你的容忍度时，才启用自动回滚。参考：https://www.flagsmith.com/blog/when-canary-alerts-go-wrong

Switchback experiments for two-sided systems

双边系统使用标准A/B测试

Standard A/B fails on marketplaces, ride-share, ad auctions, and other systems where the treatment group affects the control group through shared state. Splitting traffic 50/50 doesn't isolate the experiment — both sides see the same warped market. Fix: Use a switchback design — alternate the entire system between control and treatment over short windows (minutes to hours). Statsig's Switchback experiments (Feb 2026) automate this for the common cases. Don't block release on a corrupted A/B test result; rerun with the right design. Reference: https://www.statsig.com/updates

标准A/B测试在市场、网约车、广告拍卖等系统中失效，因为处理组会通过共享状态影响对照组。将流量分成50/50无法隔离实验——双方都会看到扭曲的市场。 修复方案： 使用开关设计——在短窗口（分钟到小时）内交替整个系统的对照组和处理组。Statsig的Switchback experiments（2026年2月）为常见场景自动化了此流程。不要因损坏的A/B测试结果而阻碍发布；使用正确的设计重新运行。参考：https://www.statsig.com/updates

Verification

验证操作

Run these immediately after the deploy completes, smallest check first. Expand the error-tracker queries in

references/rollout-automation.md

bash

undefined

部署完成后立即执行以下操作，从最小的检查开始。

references/rollout-automation.md

中有更详细的错误跟踪工具查询语句。

bash

undefined

Health endpoint returns healthy

健康端点返回健康状态

curl -s https://your-app.com/health | jq .

Response time + status in one shot

一次性获取响应时间和状态码

curl -o /dev/null -s -w "HTTP %{http_code} in %{time_total}s\n" https://your-app.com

New errors since deploy (Sentry CLI) — should be empty

部署后新增的错误（Sentry CLI）——应返回空

sentry-cli issues list --project your-project --query "firstSeen:>15m"

Datadog: compare 5xx count for the service before vs after deploy — must not increase

Datadog：对比服务部署前后的5xx错误数量——不得增加


Pass criteria: health returns `healthy`, status is `200` within your latency budget, the Sentry query returns no new issues, and the post-deploy 5xx count is at or below the pre-deploy baseline.

---


通过标准：健康端点返回`healthy`、状态码为`200`且在你的延迟预算内、Sentry查询无新增问题、部署后5xx错误数量等于或低于部署前基准线。

---

Done When

完成标准

Go/no-go checklist completed with evidence for each item and stored as a versioned artifact (e.g.
```
RELEASE-<version>.md
```
or a tracked issue), signed off by the named approver with a timestamp
Smoke test suite run against the release candidate in staging — all tests pass (exit code 0)
Rollback criteria documented as specific baseline-relative thresholds, and the rollback procedure practiced on staging at least once
Staged rollout plan defined with traffic percentages, per-stage promotion criteria, and guardrail metrics for each stage
Post-deployment verification commands run and passing (health 200, no new Sentry issues, 5xx count at or below baseline)

上线/不上线检查清单已填写完成，每项都有实证支持，并作为版本化工件存储（例如
```
RELEASE-<version>.md
```
或跟踪工单），由指定审批人签字并标注时间戳
针对预发布环境中的发布候选版本运行冒烟测试套件——所有测试通过（退出码0）
回滚标准已文档化为具体的基准线相对阈值，且回滚流程已在预发布环境至少演练一次
分阶段发布计划已定义，包含流量占比、各阶段推进标准和每个阶段的防护指标
部署后验证命令已执行并通过（健康状态200、无新增Sentry问题、5xx错误数量等于或低于基准线）

Reference Files (in

references/

)

参考文件（位于

references/

目录）

rollout-automation.md — Automated canary→10%→50%→100% promotion criteria (baseline-relative) and post-deployment verification command snippets.
communication-templates.md — Load when writing a release or rollback announcement: fill-in-the-blank release and rollback templates.

rollout-automation.md —— 自动化金丝雀→10%→50%→100%推进标准（基于基准线）和部署后验证命令片段。
communication-templates.md —— 撰写发布或回滚通知时使用：填空式的发布和回滚模板。