Working on incident response incident response tasks or workflows
Needing guidance, best practices, or checklists for incident response incident response
Do not use this skill when
The task is unrelated to incident response incident response
You need a different domain or tool outside this scope
Instructions
Clarify goals, constraints, and required inputs.
Apply relevant best practices and validate outcomes.
Provide actionable steps and verification.
If detailed examples are required, open
resources/implementation-playbook.md
.
Orchestrate multi-agent incident response with modern SRE practices for rapid resolution and learning:
[Extended thinking: This workflow implements a comprehensive incident command system (ICS) following modern SRE principles. Multiple specialized agents collaborate through defined phases: detection/triage, investigation/mitigation, communication/coordination, and resolution/postmortem. The workflow emphasizes speed without sacrificing accuracy, maintains clear communication channels, and ensures every incident becomes a learning opportunity through blameless postmortems and systematic improvements.]
Configuration
Severity Levels
P0/SEV-1: Complete outage, security breach, data loss - immediate all-hands response
P1/SEV-2: Major degradation, significant user impact - rapid response required
P2/SEV-3: Minor degradation, limited impact - standard response
P3/SEV-4: Cosmetic issues, no user impact - scheduled resolution
Incident Types
Performance degradation
Service outage
Security incident
Data integrity issue
Infrastructure failure
Third-party service disruption
Phase 1: Detection & Triage
1. Incident Detection and Classification
Use Task tool with subagent_type="incident-responder"
Prompt: "URGENT: Detect and classify incident: $ARGUMENTS. Analyze alerts from PagerDuty/Opsgenie/monitoring. Determine: 1) Incident severity (P0-P3), 2) Affected services and dependencies, 3) User impact and business risk, 4) Initial incident command structure needed. Check error budgets and SLO violations."
Output: Severity classification, impact assessment, incident command assignments, SLO status
Use Task tool with subagent_type="error-debugging::debugger"
Prompt: "Conduct deep debugging for incident: $ARGUMENTS using observability data. Investigate: 1) Stack traces and error logs, 2) Database query performance and locks, 3) Network latency and timeouts, 4) Memory leaks and CPU spikes, 5) Dependency failures and cascading errors. Apply Five Whys analysis."
Output: Root cause identification, contributing factors, dependency impact map
Context: Observability analysis, mitigation status
5. Security Assessment
Use Task tool with subagent_type="security-scanning::security-auditor"
Use Task tool with subagent_type="backend-development::backend-architect"
Prompt: "Design and implement production fix for incident: $ARGUMENTS based on root cause. Requirements: 1) Minimal viable fix for rapid deployment, 2) Risk assessment and rollback capability, 3) Staged rollout plan with monitoring, 4) Validation criteria and health checks. Consider both immediate fix and long-term solution."
Context: Root cause analysis, performance findings, security assessment
8. Deployment and Validation
Use Task tool with subagent_type="deployment-strategies::deployment-engineer"
Prompt: "Execute emergency deployment for incident fix: $ARGUMENTS. Process: 1) Blue-green or canary deployment, 2) Progressive rollout with monitoring, 3) Health check validation at each stage, 4) Rollback triggers configured, 5) Real-time monitoring during deployment. Coordinate with incident command."
Context: Resolution progress, communication status
Phase 5: Postmortem & Prevention
11. Blameless Postmortem
Use Task tool with subagent_type="documentation-generation::docs-architect"
Prompt: "Conduct blameless postmortem for incident: $ARGUMENTS. Document: 1) Complete incident timeline with decisions, 2) Root cause and contributing factors (systems focus), 3) What went well in response, 4) What could improve, 5) Action items with owners and deadlines, 6) Lessons learned for team education. Follow SRE postmortem best practices."
Output: Postmortem document, action items list, process improvements, training needs
Context: Complete incident history, all agent outputs
12. Monitoring and Alert Enhancement
Use Task tool with subagent_type="observability-monitoring::observability-engineer"
Prompt: "Enhance monitoring to prevent recurrence of: $ARGUMENTS. Implement: 1) New alerts for early detection, 2) SLI/SLO adjustments if needed, 3) Dashboard improvements for visibility, 4) Runbook automation opportunities, 5) Chaos engineering scenarios for testing. Ensure alerts are actionable and reduce noise."
Output: New monitoring configuration, alert rules, dashboard updates, runbook automation
Context: Postmortem findings, root cause analysis
13. System Hardening
Use Task tool with subagent_type="backend-development::backend-architect"