Incident Response

事故响应

Handle production incidents systematically.

系统化处理生产环境事故。

When to Use

适用场景

Production is down or degraded
Critical errors affecting users
Security incidents
Data issues
Performance emergencies

生产环境宕机或性能下降
影响用户的严重错误
安全事件
数据问题
性能紧急事件

Incident Workflow

事故处理流程

DETECT → TRIAGE → MITIGATE → RESOLVE → REVIEW

DETECT → TRIAGE → MITIGATE → RESOLVE → REVIEW

1. Detect & Triage

1. 检测与分级

bash

undefined

bash

undefined

Quick health checks

curl -s https://api.example.com/health | jq . kubectl get pods -n production | grep -v Running

Check recent deployments

git log --oneline -5 kubectl rollout history deployment/app

Error rates

grep -c "ERROR" /var/log/app.log

undefined

grep -c "ERROR" /var/log/app.log

undefined

2. Mitigate First

2. 优先缓解

Priority: Stop the bleeding before finding root cause

bash

undefined

优先级：在找到根本原因前先止损

bash

undefined

Rollback deployment

kubectl rollout undo deployment/app

Scale up if overloaded

kubectl scale deployment/app --replicas=10

Feature flag disable

curl -X POST api.example.com/admin/flags -d '{"feature": false}'

Circuit breaker

Block problematic endpoint or dependency

undefined

undefined

3. Investigate

3. 调查分析

bash

undefined

bash

undefined

Recent logs

kubectl logs -l app=myapp --since=30m | grep -i error

Resource usage

kubectl top pods -n production

Database connections

SELECT count(*) FROM pg_stat_activity WHERE state = 'active';

Network issues

curl -w "@curl-format.txt" -o /dev/null -s https://api.example.com

undefined

curl -w "@curl-format.txt" -o /dev/null -s https://api.example.com

undefined

Severity Levels

严重级别

Level	Impact	Response Time	Example
P1	Complete outage	Immediate	Site down
P2	Major feature broken	15 min	Payments failing
P3	Minor feature broken	1 hour	Search slow
P4	Low impact	Next day	UI glitch

级别	影响范围	响应时间	示例
P1	完全宕机	立即响应	站点无法访问
P2	核心功能故障	15分钟内响应	支付功能失效
P3	次要功能故障	1小时内响应	搜索功能缓慢
P4	低影响	次日响应	UI界面小问题

Communication Template

沟通模板

markdown

undefined

markdown

undefined

Incident Update

Status: Investigating | Identified | Mitigated | Resolved Severity: P1/P2/P3 Started: YYYY-MM-DD HH:MM UTC Duration: X hours

Summary

[1-2 sentences on what's happening]

Impact

[Who is affected and how]

Current Actions

[Action 1]
[Action 2]

[Action 1]
[Action 2]

Next Update

[Time of next update]

undefined

[Time of next update]

undefined

Post-Mortem Template

事后复盘模板

markdown

undefined

markdown

undefined

Incident Post-Mortem

Date: YYYY-MM-DD Duration: X hours Severity: P1

Summary

[What happened in 2-3 sentences]

Timeline

HH:MM - [Event]
HH:MM - [Event]

HH:MM - [Event]
HH:MM - [Event]

Root Cause

[Technical explanation]

Impact

Users affected: X
Revenue impact: $Y
Data loss: None/Describe

Users affected: X
Revenue impact: $Y
Data loss: None/Describe

Action Items

Action	Owner	Due Date
Add monitoring for X	@name	YYYY-MM-DD
Improve circuit breaker	@name	YYYY-MM-DD

Action	Owner	Due Date
Add monitoring for X	@name	YYYY-MM-DD
Improve circuit breaker	@name	YYYY-MM-DD

Lessons Learned

[What we learned]

undefined

[What we learned]

undefined

Examples

示例

Input: "API is returning 500 errors" Action: Check logs, identify failing component, rollback if recent deploy, fix

Input: "Database is overloaded" Action: Kill long queries, scale read replicas, optimize or cache hot queries

输入： "API返回500错误" 行动： 查看日志，定位故障组件，若为近期部署则回滚，修复问题

输入： "数据库负载过高" 行动： 终止长查询，扩容只读副本，优化或缓存热点查询

incident

Original

Translation

Incident Response

事故响应

When to Use

适用场景

Incident Workflow

事故处理流程

1. Detect & Triage

1. 检测与分级

Quick health checks

Quick health checks

Check recent deployments

Check recent deployments

Error rates

Error rates

2. Mitigate First

2. 优先缓解

Rollback deployment

Rollback deployment

Scale up if overloaded

Scale up if overloaded

Feature flag disable

Feature flag disable

Circuit breaker

Circuit breaker

Block problematic endpoint or dependency

Block problematic endpoint or dependency

3. Investigate

3. 调查分析

Recent logs

Recent logs

Resource usage

Resource usage

Database connections

Database connections

Network issues

Network issues

Severity Levels

严重级别

Communication Template

沟通模板

Incident Update

Incident Update

Summary

Summary

Impact

Impact

Current Actions

Current Actions

Next Update

Next Update

Post-Mortem Template

事后复盘模板

Incident Post-Mortem

Incident Post-Mortem

Summary

Summary

Timeline

Timeline

Root Cause

Root Cause

Impact

Impact

Action Items

Action Items

Lessons Learned

Lessons Learned

Examples

示例