Back to Details

incident-response-incident-response

Compare original and translation side by side

🇺🇸

Original

English
🇨🇳

Translation

Chinese

Use this skill when

适用本Skill的场景

  • Working on incident response incident response tasks or workflows
  • Needing guidance, best practices, or checklists for incident response incident response
  • 处理事件响应相关任务或工作流时
  • 需要事件响应的指导、最佳实践或检查清单时

Do not use this skill when

不适用本Skill的场景

  • The task is unrelated to incident response incident response
  • You need a different domain or tool outside this scope
  • 任务与事件响应无关时
  • 需要该范围之外的其他领域或工具时

Instructions

操作指引

  • Clarify goals, constraints, and required inputs.
  • Apply relevant best practices and validate outcomes.
  • Provide actionable steps and verification.
  • If detailed examples are required, open 
    resources/implementation-playbook.md
    .
Orchestrate multi-agent incident response with modern SRE practices for rapid resolution and learning:
[Extended thinking: This workflow implements a comprehensive incident command system (ICS) following modern SRE principles. Multiple specialized agents collaborate through defined phases: detection/triage, investigation/mitigation, communication/coordination, and resolution/postmortem. The workflow emphasizes speed without sacrificing accuracy, maintains clear communication channels, and ensures every incident becomes a learning opportunity through blameless postmortems and systematic improvements.]
  • 明确目标、约束条件和所需输入。
  • 应用相关最佳实践并验证结果。
  • 提供可执行步骤和验证方法。
  • 如果需要详细示例，请打开
    resources/implementation-playbook.md
    。
结合现代SRE实践编排多Agent事件响应流程，实现快速问题解决与经验沉淀：
[扩展思考：该工作流遵循现代SRE原则，实现了一套全面的事件指挥系统（ICS）。多个专业Agent通过定义好的阶段协作：检测/分类、调查/缓解、沟通/协调、解决/事后复盘。该工作流强调在不牺牲准确性的前提下提升速度，保持清晰的沟通渠道，并通过无责事后复盘和系统性改进，确保每一起事件都成为学习机会。]

Configuration

配置说明

Severity Levels

严重级别

  • P0/SEV-1: Complete outage, security breach, data loss - immediate all-hands response
  • P1/SEV-2: Major degradation, significant user impact - rapid response required
  • P2/SEV-3: Minor degradation, limited impact - standard response
  • P3/SEV-4: Cosmetic issues, no user impact - scheduled resolution
  • P0/SEV-1：完全中断、安全漏洞、数据丢失 - 立即全员响应
  • P1/SEV-2：严重性能下降、显著用户影响 - 需要快速响应
  • P2/SEV-3：轻微性能下降、影响有限 - 标准响应流程
  • P3/SEV-4：外观问题、无用户影响 - 排期解决

Incident Types

事件类型

  • Performance degradation
  • Service outage
  • Security incident
  • Data integrity issue
  • Infrastructure failure
  • Third-party service disruption
  • 性能下降
  • 服务中断
  • 安全事件
  • 数据完整性问题
  • 基础设施故障
  • 第三方服务中断

Phase 1: Detection & Triage

第一阶段：检测与分类

1. Incident Detection and Classification

1. 事件检测与分类

  • Use Task tool with subagent_type="incident-responder"
  • Prompt: "URGENT: Detect and classify incident: $ARGUMENTS. Analyze alerts from PagerDuty/Opsgenie/monitoring. Determine: 1) Incident severity (P0-P3), 2) Affected services and dependencies, 3) User impact and business risk, 4) Initial incident command structure needed. Check error budgets and SLO violations."
  • Output: Severity classification, impact assessment, incident command assignments, SLO status
  • Context: Initial alerts, monitoring dashboards, recent changes
  • 使用Task tool，设置subagent_type="incident-responder"
  • 提示语："URGENT: Detect and classify incident: $ARGUMENTS. Analyze alerts from PagerDuty/Opsgenie/monitoring. Determine: 1) Incident severity (P0-P3), 2) Affected services and dependencies, 3) User impact and business risk, 4) Initial incident command structure needed. Check error budgets and SLO violations."
  • 输出：严重级别分类、影响评估、事件指挥角色分配、SLO状态
  • 上下文：初始告警、监控仪表盘、近期变更

2. Observability Analysis

2. 可观测性分析

  • Use Task tool with subagent_type="observability-monitoring::observability-engineer"
  • Prompt: "Perform rapid observability sweep for incident: $ARGUMENTS. Query: 1) Distributed tracing (OpenTelemetry/Jaeger), 2) Metrics correlation (Prometheus/Grafana/DataDog), 3) Log aggregation (ELK/Splunk), 4) APM data, 5) Real User Monitoring. Identify anomalies, error patterns, and service degradation points."
  • Output: Observability findings, anomaly detection, service health matrix, trace analysis
  • Context: Severity level from step 1, affected services
  • 使用Task tool，设置subagent_type="observability-monitoring::observability-engineer"
  • 提示语："Perform rapid observability sweep for incident: $ARGUMENTS. Query: 1) Distributed tracing (OpenTelemetry/Jaeger), 2) Metrics correlation (Prometheus/Grafana/DataDog), 3) Log aggregation (ELK/Splunk), 4) APM data, 5) Real User Monitoring. Identify anomalies, error patterns, and service degradation points."
  • 输出：可观测性分析结果、异常检测报告、服务健康矩阵、链路追踪分析
  • 上下文：步骤1得出的严重级别、受影响服务

3. Initial Mitigation

3. 初步缓解

  • Use Task tool with subagent_type="incident-responder"
  • Prompt: "Implement immediate mitigation for P$SEVERITY incident: $ARGUMENTS. Actions: 1) Traffic throttling/rerouting if needed, 2) Feature flag disabling for affected features, 3) Circuit breaker activation, 4) Rollback assessment for recent deployments, 5) Scale resources if capacity-related. Prioritize user experience restoration."
  • Output: Mitigation actions taken, temporary fixes applied, rollback decisions
  • Context: Observability findings, severity classification
  • 使用Task tool，设置subagent_type="incident-responder"
  • 提示语："Implement immediate mitigation for P$SEVERITY incident: $ARGUMENTS. Actions: 1) Traffic throttling/rerouting if needed, 2) Feature flag disabling for affected features, 3) Circuit breaker activation, 4) Rollback assessment for recent deployments, 5) Scale resources if capacity-related. Prioritize user experience restoration."
  • 输出：已执行的缓解措施、临时修复方案、回滚决策
  • 上下文：可观测性分析结果、严重级别分类

Phase 2: Investigation & Root Cause Analysis

第二阶段：调查与根因分析

4. Deep System Debugging

4. 深度系统调试

  • Use Task tool with subagent_type="error-debugging::debugger"
  • Prompt: "Conduct deep debugging for incident: $ARGUMENTS using observability data. Investigate: 1) Stack traces and error logs, 2) Database query performance and locks, 3) Network latency and timeouts, 4) Memory leaks and CPU spikes, 5) Dependency failures and cascading errors. Apply Five Whys analysis."
  • Output: Root cause identification, contributing factors, dependency impact map
  • Context: Observability analysis, mitigation status
  • 使用Task tool，设置subagent_type="error-debugging::debugger"
  • 提示语："Conduct deep debugging for incident: $ARGUMENTS using observability data. Investigate: 1) Stack traces and error logs, 2) Database query performance and locks, 3) Network latency and timeouts, 4) Memory leaks and CPU spikes, 5) Dependency failures and cascading errors. Apply Five Whys analysis."
  • 输出：根因定位结果、影响因素分析、依赖关系影响图
  • 上下文：可观测性分析结果、缓解措施执行状态

5. Security Assessment

5. 安全评估

  • Use Task tool with subagent_type="security-scanning::security-auditor"
  • Prompt: "Assess security implications of incident: $ARGUMENTS. Check: 1) DDoS attack indicators, 2) Authentication/authorization failures, 3) Data exposure risks, 4) Certificate issues, 5) Suspicious access patterns. Review WAF logs, security groups, and audit trails."
  • Output: Security assessment, breach analysis, vulnerability identification
  • Context: Root cause findings, system logs
  • 使用Task tool，设置subagent_type="security-scanning::security-auditor"
  • 提示语："Assess security implications of incident: $ARGUMENTS. Check: 1) DDoS attack indicators, 2) Authentication/authorization failures, 3) Data exposure risks, 4) Certificate issues, 5) Suspicious access patterns. Review WAF logs, security groups, and audit trails."
  • 输出：安全评估报告、漏洞分析结果、风险点识别
  • 上下文：根因分析结果、系统日志

6. Performance Engineering Analysis

6. 性能工程分析

  • Use Task tool with subagent_type="application-performance::performance-engineer"
  • Prompt: "Analyze performance aspects of incident: $ARGUMENTS. Examine: 1) Resource utilization patterns, 2) Query optimization opportunities, 3) Caching effectiveness, 4) Load balancer health, 5) CDN performance, 6) Autoscaling triggers. Identify bottlenecks and capacity issues."
  • Output: Performance bottlenecks, resource recommendations, optimization opportunities
  • Context: Debug findings, current mitigation state
  • 使用Task tool，设置subagent_type="application-performance::performance-engineer"
  • 提示语："Analyze performance aspects of incident: $ARGUMENTS. Examine: 1) Resource utilization patterns, 2) Query optimization opportunities, 3) Caching effectiveness, 4) Load balancer health, 5) CDN performance, 6) Autoscaling triggers. Identify bottlenecks and capacity issues."
  • 输出：性能瓶颈定位、资源优化建议、性能提升方向
  • 上下文：调试结果、当前缓解状态

Phase 3: Resolution & Recovery

第三阶段：问题解决与恢复

7. Fix Implementation

7. 修复方案实施

  • Use Task tool with subagent_type="backend-development::backend-architect"
  • Prompt: "Design and implement production fix for incident: $ARGUMENTS based on root cause. Requirements: 1) Minimal viable fix for rapid deployment, 2) Risk assessment and rollback capability, 3) Staged rollout plan with monitoring, 4) Validation criteria and health checks. Consider both immediate fix and long-term solution."
  • Output: Fix implementation, deployment strategy, validation plan, rollback procedures
  • Context: Root cause analysis, performance findings, security assessment
  • 使用Task tool，设置subagent_type="backend-development::backend-architect"
  • 提示语："Design and implement production fix for incident: $ARGUMENTS based on root cause. Requirements: 1) Minimal viable fix for rapid deployment, 2) Risk assessment and rollback capability, 3) Staged rollout plan with monitoring, 4) Validation criteria and health checks. Consider both immediate fix and long-term solution."
  • 输出：修复方案、部署策略、验证计划、回滚流程
  • 上下文：根因分析结果、性能分析结果、安全评估结果

8. Deployment and Validation

8. 部署与验证

  • Use Task tool with subagent_type="deployment-strategies::deployment-engineer"
  • Prompt: "Execute emergency deployment for incident fix: $ARGUMENTS. Process: 1) Blue-green or canary deployment, 2) Progressive rollout with monitoring, 3) Health check validation at each stage, 4) Rollback triggers configured, 5) Real-time monitoring during deployment. Coordinate with incident command."
  • Output: Deployment status, validation results, monitoring dashboard, rollback readiness
  • Context: Fix implementation, current system state
  • 使用Task tool，设置subagent_type="deployment-strategies::deployment-engineer"
  • 提示语："Execute emergency deployment for incident fix: $ARGUMENTS. Process: 1) Blue-green or canary deployment, 2) Progressive rollout with monitoring, 3) Health check validation at each stage, 4) Rollback triggers configured, 5) Real-time monitoring during deployment. Coordinate with incident command."
  • 输出：部署状态、验证结果、监控仪表盘、回滚就绪状态
  • 上下文：修复方案、当前系统状态

Phase 4: Communication & Coordination

第四阶段：沟通与协调

9. Stakeholder Communication

9. 利益相关方沟通

  • Use Task tool with subagent_type="content-marketing::content-marketer"
  • Prompt: "Manage incident communication for: $ARGUMENTS. Create: 1) Status page updates (public-facing), 2) Internal engineering updates (technical details), 3) Executive summary (business impact/ETA), 4) Customer support briefing (talking points), 5) Timeline documentation with key decisions. Update every 15-30 minutes based on severity."
  • Output: Communication artifacts, status updates, stakeholder briefings, timeline log
  • Context: All previous phases, current resolution status
  • 使用Task tool，设置subagent_type="content-marketing::content-marketer"
  • 提示语："Manage incident communication for: $ARGUMENTS. Create: 1) Status page updates (public-facing), 2) Internal engineering updates (technical details), 3) Executive summary (business impact/ETA), 4) Customer support briefing (talking points), 5) Timeline documentation with key decisions. Update every 15-30 minutes based on severity."
  • 输出：沟通文档、状态更新、利益相关方简报、时间线日志
  • 上下文：所有前期阶段内容、当前解决状态

10. Customer Impact Assessment

10. 用户影响评估

  • Use Task tool with subagent_type="incident-responder"
  • Prompt: "Assess and document customer impact for incident: $ARGUMENTS. Analyze: 1) Affected user segments and geography, 2) Failed transactions or data loss, 3) SLA violations and contractual implications, 4) Customer support ticket volume, 5) Revenue impact estimation. Prepare proactive customer outreach list."
  • Output: Customer impact report, SLA analysis, outreach recommendations
  • Context: Resolution progress, communication status
  • 使用Task tool，设置subagent_type="incident-responder"
  • 提示语："Assess and document customer impact for incident: $ARGUMENTS. Analyze: 1) Affected user segments and geography, 2) Failed transactions or data loss, 3) SLA violations and contractual implications, 4) Customer support ticket volume, 5) Revenue impact estimation. Prepare proactive customer outreach list."
  • 输出：用户影响报告、SLA合规性分析、主动沟通建议
  • 上下文：解决进度、沟通状态

Phase 5: Postmortem & Prevention

第五阶段：事后复盘与预防

11. Blameless Postmortem

11. 无责事后复盘

  • Use Task tool with subagent_type="documentation-generation::docs-architect"
  • Prompt: "Conduct blameless postmortem for incident: $ARGUMENTS. Document: 1) Complete incident timeline with decisions, 2) Root cause and contributing factors (systems focus), 3) What went well in response, 4) What could improve, 5) Action items with owners and deadlines, 6) Lessons learned for team education. Follow SRE postmortem best practices."
  • Output: Postmortem document, action items list, process improvements, training needs
  • Context: Complete incident history, all agent outputs
  • 使用Task tool，设置subagent_type="documentation-generation::docs-architect"
  • 提示语："Conduct blameless postmortem for incident: $ARGUMENTS. Document: 1) Complete incident timeline with decisions, 2) Root cause and contributing factors (systems focus), 3) What went well in response, 4) What could improve, 5) Action items with owners and deadlines, 6) Lessons learned for team education. Follow SRE postmortem best practices."
  • 输出：事后复盘文档、行动项列表、流程改进建议、培训需求
  • 上下文：完整事件历史、所有Agent输出内容

12. Monitoring and Alert Enhancement

12. 监控与告警优化

  • Use Task tool with subagent_type="observability-monitoring::observability-engineer"
  • Prompt: "Enhance monitoring to prevent recurrence of: $ARGUMENTS. Implement: 1) New alerts for early detection, 2) SLI/SLO adjustments if needed, 3) Dashboard improvements for visibility, 4) Runbook automation opportunities, 5) Chaos engineering scenarios for testing. Ensure alerts are actionable and reduce noise."
  • Output: New monitoring configuration, alert rules, dashboard updates, runbook automation
  • Context: Postmortem findings, root cause analysis
  • 使用Task tool，设置subagent_type="observability-monitoring::observability-engineer"
  • 提示语："Enhance monitoring to prevent recurrence of: $ARGUMENTS. Implement: 1) New alerts for early detection, 2) SLI/SLO adjustments if needed, 3) Dashboard improvements for visibility, 4) Runbook automation opportunities, 5) Chaos engineering scenarios for testing. Ensure alerts are actionable and reduce noise."
  • 输出：新监控配置、告警规则、仪表盘更新、自动化手册
  • 上下文：事后复盘结果、根因分析

13. System Hardening

13. 系统加固

  • Use Task tool with subagent_type="backend-development::backend-architect"
  • Prompt: "Design system improvements to prevent incident: $ARGUMENTS. Propose: 1) Architecture changes for resilience (circuit breakers, bulkheads), 2) Graceful degradation strategies, 3) Capacity planning adjustments, 4) Technical debt prioritization, 5) Dependency reduction opportunities. Create implementation roadmap."
  • Output: Architecture improvements, resilience patterns, technical debt items, roadmap
  • Context: Postmortem action items, performance analysis
  • 使用Task tool，设置subagent_type="backend-development::backend-architect"
  • 提示语："Design system improvements to prevent incident: $ARGUMENTS. Propose: 1) Architecture changes for resilience (circuit breakers, bulkheads), 2) Graceful degradation strategies, 3) Capacity planning adjustments, 4) Technical debt prioritization, 5) Dependency reduction opportunities. Create implementation roadmap."
  • 输出：架构优化方案、弹性设计模式、技术债务清单、实施路线图
  • 上下文：事后复盘行动项、性能分析结果

Success Criteria

成功标准

Immediate Success (During Incident)

即时成功（事件处理期间）

  • Service restoration within SLA targets
  • Accurate severity classification within 5 minutes
  • Stakeholder communication every 15-30 minutes
  • No cascading failures or incident escalation
  • Clear incident command structure maintained
  • 在SLA目标内恢复服务
  • 5分钟内完成准确的严重级别分类
  • 每15-30分钟向利益相关方同步状态
  • 无连锁故障或事件升级
  • 保持清晰的事件指挥架构

Long-term Success (Post-Incident)

长期成功（事件处理后）

  • Comprehensive postmortem within 48 hours
  • All action items assigned with deadlines
  • Monitoring improvements deployed within 1 week
  • Runbook updates completed
  • Team training conducted on lessons learned
  • Error budget impact assessed and communicated
  • 48小时内完成全面的事后复盘
  • 所有行动项分配责任人与截止日期
  • 1周内部署监控优化措施
  • 完成操作手册更新
  • 针对经验教训开展团队培训
  • 评估并沟通错误预算影响

Coordination Protocols

协调协议

Incident Command Structure

事件指挥架构

  • Incident Commander: Decision authority, coordination
  • Technical Lead: Technical investigation and resolution
  • Communications Lead: Stakeholder updates
  • Subject Matter Experts: Specific system expertise
  • 事件指挥官：决策负责人、全局协调
  • 技术负责人：技术调查与问题解决
  • 沟通负责人：利益相关方状态更新
  • 领域专家：特定系统专业支持

Communication Channels

沟通渠道

  • War room (Slack/Teams channel or Zoom)
  • Status page updates (StatusPage, Statusly)
  • PagerDuty/Opsgenie for alerting
  • Confluence/Notion for documentation
  • 作战室（Slack/Teams频道或Zoom会议）
  • 状态页面更新（StatusPage、Statusly）
  • PagerDuty/Opsgenie告警
  • Confluence/Notion文档记录

Handoff Requirements

交接要求

  • Each phase provides clear context to the next
  • All findings documented in shared incident doc
  • Decision rationale recorded for postmortem
  • Timestamp all significant events
Production incident requiring immediate response: $ARGUMENTS
  • 每个阶段向下一阶段提供清晰上下文
  • 所有分析结果记录在共享事件文档中
  • 决策依据需记录用于事后复盘
  • 所有重要事件需标记时间戳
需要立即响应的生产事件：$ARGUMENTS