alerting-oncall

Compare original and translation side by side

🇺🇸

Original

English

🇨🇳

Translation

Chinese

Alerting & On-Call

告警与值班管理

Configure effective alerting and on-call management for production systems.

为生产系统配置有效的告警和值班管理。

When to Use This Skill

何时使用此技能

Use this skill when:

Setting up alerting rules and thresholds
Configuring on-call rotations and schedules
Implementing alert routing and escalation
Reducing alert fatigue
Managing incident response workflows

在以下场景使用此技能：

设置告警规则和阈值
配置值班轮换和排班
实施告警路由和升级
减少告警疲劳
管理事件响应工作流

Prerequisites

前提条件

Monitoring system (Prometheus, Datadog, etc.)
On-call platform (PagerDuty, Opsgenie, Grafana OnCall)
Communication channels (Slack, email)

监控系统（Prometheus、Datadog等）
值班平台（PagerDuty、Opsgenie、Grafana OnCall）
沟通渠道（Slack、邮件）

Alerting Best Practices

告警最佳实践

Alert Categories

告警分类

yaml

undefined

yaml

undefined

Severity levels

critical:

Service completely down
Data loss imminent
Security breach response: Immediate page, wake people up

high:

Service degraded significantly
Error rate above SLO
Capacity near limit response: Page during business hours, notify after hours

medium:

Performance degradation
Non-critical component failure
Warning thresholds exceeded response: Notify via Slack, review next business day

low:

Informational alerts
Capacity planning triggers
Routine maintenance needed response: Email notification, weekly review

undefined

critical:

Service completely down
Data loss imminent
Security breach response: Immediate page, wake people up

high:

Service degraded significantly
Error rate above SLO
Capacity near limit response: Page during business hours, notify after hours

medium:

Performance degradation
Non-critical component failure
Warning thresholds exceeded response: Notify via Slack, review next business day

low:

Informational alerts
Capacity planning triggers
Routine maintenance needed response: Email notification, weekly review

undefined

Alert Design Principles

告警设计原则

yaml

undefined

yaml

undefined

Good alert characteristics

alerts: actionable: - Every alert should require human action - Include runbook links - Clear remediation steps

relevant: - Alert on symptoms, not causes - Focus on user impact - Avoid alerting on expected behavior

timely: - Appropriate thresholds - Suitable evaluation windows - Account for normal variance

unique: - No duplicate alerts - Proper alert grouping - Clear ownership

undefined

alerts: actionable: - Every alert should require human action - Include runbook links - Clear remediation steps

relevant: - Alert on symptoms, not causes - Focus on user impact - Avoid alerting on expected behavior

timely: - Appropriate thresholds - Suitable evaluation windows - Account for normal variance

unique: - No duplicate alerts - Proper alert grouping - Clear ownership

undefined

Prometheus Alerting

Prometheus告警配置

Alert Rules

告警规则

yaml

undefined

yaml

undefined

prometheus/rules/alerts.yml

groups:

name: service_alerts rules:

High-level service health
- alert: ServiceDown expr: up{job="myapp"} == 0 for: 1m labels: severity: critical annotations: summary: "Service {{ $labels.instance }} is down" description: "{{ $labels.job }} on {{ $labels.instance }} has been down for more than 1 minute." runbook_url: "https://wiki.example.com/runbooks/service-down"
Error rate alert
- alert: HighErrorRate expr: | sum(rate(http_requests_total{status=~"5.."}[5m])) by (service) / sum(rate(http_requests_total[5m])) by (service) > 0.05 for: 5m labels: severity: critical annotations: summary: "High error rate for {{ $labels.service }}" description: "Error rate is {{ $value | humanizePercentage }} for the last 5 minutes"
Latency alert (SLO-based)
- alert: HighLatency expr: | histogram_quantile(0.95, sum(rate(http_request_duration_seconds_bucket[5m])) by (le, service) ) > 0.5 for: 5m labels: severity: high annotations: summary: "P95 latency above 500ms for {{ $labels.service }}"

undefined

groups:

name: service_alerts rules:

High-level service health
- alert: ServiceDown expr: up{job="myapp"} == 0 for: 1m labels: severity: critical annotations: summary: "Service {{ $labels.instance }} is down" description: "{{ $labels.job }} on {{ $labels.instance }} has been down for more than 1 minute." runbook_url: "https://wiki.example.com/runbooks/service-down"
Error rate alert
- alert: HighErrorRate expr: | sum(rate(http_requests_total{status=~"5.."}[5m])) by (service) / sum(rate(http_requests_total[5m])) by (service) > 0.05 for: 5m labels: severity: critical annotations: summary: "High error rate for {{ $labels.service }}" description: "Error rate is {{ $value | humanizePercentage }} for the last 5 minutes"
Latency alert (SLO-based)
- alert: HighLatency expr: | histogram_quantile(0.95, sum(rate(http_request_duration_seconds_bucket[5m])) by (le, service) ) > 0.5 for: 5m labels: severity: high annotations: summary: "P95 latency above 500ms for {{ $labels.service }}"

undefined

Alertmanager Configuration

Alertmanager配置

yaml

undefined

yaml

undefined

alertmanager.yml

global: resolve_timeout: 5m slack_api_url: 'https://hooks.slack.com/services/xxx' pagerduty_url: 'https://events.pagerduty.com/v2/enqueue'

templates:

'/etc/alertmanager/templates/*.tmpl'

route: receiver: 'default-receiver' group_by: ['alertname', 'service'] group_wait: 30s group_interval: 5m repeat_interval: 4h

routes: # Critical alerts go to PagerDuty - match: severity: critical receiver: 'pagerduty-critical' group_wait: 0s repeat_interval: 1h

# High severity during business hours
- match:
    severity: high
  receiver: 'slack-high'
  active_time_intervals:
    - business-hours

# Route by team
- match_re:
    team: platform.*
  receiver: 'platform-team'

receivers:

name: 'default-receiver' slack_configs:
- channel: '#alerts' send_resolved: true
name: 'pagerduty-critical' pagerduty_configs:
- service_key: 'xxx' severity: critical description: '{{ .CommonAnnotations.summary }}' details: firing: '{{ template "pagerduty.firing" . }}'
name: 'slack-high' slack_configs:
- channel: '#alerts-high' title: '{{ .CommonAnnotations.summary }}' text: '{{ .CommonAnnotations.description }}' actions:
  - type: button text: 'Runbook' url: '{{ .CommonAnnotations.runbook_url }}'
  - type: button text: 'Dashboard' url: '{{ .CommonAnnotations.dashboard_url }}'
name: 'platform-team' slack_configs:
- channel: '#platform-alerts'

time_intervals:

name: business-hours time_intervals:
- weekdays: ['monday:friday'] times:
  - start_time: '09:00' end_time: '17:00'

inhibit_rules:

source_match: severity: critical target_match: severity: high equal: ['service']

undefined

global: resolve_timeout: 5m slack_api_url: 'https://hooks.slack.com/services/xxx' pagerduty_url: 'https://events.pagerduty.com/v2/enqueue'

templates:

'/etc/alertmanager/templates/*.tmpl'

route: receiver: 'default-receiver' group_by: ['alertname', 'service'] group_wait: 30s group_interval: 5m repeat_interval: 4h

routes: # Critical alerts go to PagerDuty - match: severity: critical receiver: 'pagerduty-critical' group_wait: 0s repeat_interval: 1h

# High severity during business hours
- match:
    severity: high
  receiver: 'slack-high'
  active_time_intervals:
    - business-hours

# Route by team
- match_re:
    team: platform.*
  receiver: 'platform-team'

receivers:

name: 'default-receiver' slack_configs:
- channel: '#alerts' send_resolved: true
name: 'pagerduty-critical' pagerduty_configs:
- service_key: 'xxx' severity: critical description: '{{ .CommonAnnotations.summary }}' details: firing: '{{ template "pagerduty.firing" . }}'
name: 'slack-high' slack_configs:
- channel: '#alerts-high' title: '{{ .CommonAnnotations.summary }}' text: '{{ .CommonAnnotations.description }}' actions:
  - type: button text: 'Runbook' url: '{{ .CommonAnnotations.runbook_url }}'
  - type: button text: 'Dashboard' url: '{{ .CommonAnnotations.dashboard_url }}'
name: 'platform-team' slack_configs:
- channel: '#platform-alerts'

time_intervals:

name: business-hours time_intervals:
- weekdays: ['monday:friday'] times:
  - start_time: '09:00' end_time: '17:00'

inhibit_rules:

source_match: severity: critical target_match: severity: high equal: ['service']

undefined

PagerDuty Integration

PagerDuty集成

Service Configuration

服务配置

yaml

undefined

yaml

undefined

Terraform example

resource "pagerduty_service" "myapp" { name = "MyApp Production" description = "Production application service" escalation_policy = pagerduty_escalation_policy.default.id alert_creation = "create_alerts_and_incidents" auto_resolve_timeout = 14400 # 4 hours acknowledgement_timeout = 600 # 10 minutes

incident_urgency_rule { type = "use_support_hours"

during_support_hours {
  type    = "constant"
  urgency = "high"
}

outside_support_hours {
  type    = "constant"
  urgency = "low"
}

} }

resource "pagerduty_escalation_policy" "default" { name = "Default Escalation" num_loops = 2

rule { escalation_delay_in_minutes = 10 target { type = "schedule_reference" id = pagerduty_schedule.primary.id } }

rule { escalation_delay_in_minutes = 15 target { type = "user_reference" id = pagerduty_user.manager.id } } }

undefined

incident_urgency_rule { type = "use_support_hours"

during_support_hours {
  type    = "constant"
  urgency = "high"
}

outside_support_hours {
  type    = "constant"
  urgency = "low"
}

} }

resource "pagerduty_escalation_policy" "default" { name = "Default Escalation" num_loops = 2

rule { escalation_delay_in_minutes = 10 target { type = "schedule_reference" id = pagerduty_schedule.primary.id } }

rule { escalation_delay_in_minutes = 15 target { type = "user_reference" id = pagerduty_user.manager.id } } }

undefined

Schedule Configuration

排班配置

yaml

resource "pagerduty_schedule" "primary" {
  name      = "Primary On-Call"
  time_zone = "America/New_York"

  layer {
    name                         = "Weekly Rotation"
    start                        = "2024-01-01T00:00:00-05:00"
    rotation_virtual_start       = "2024-01-01T00:00:00-05:00"
    rotation_turn_length_seconds = 604800  # 1 week
    users                        = [for user in pagerduty_user.oncall : user.id]
  }

  # Override layer for holidays
  layer {
    name                         = "Holiday Coverage"
    start                        = "2024-01-01T00:00:00-05:00"
    rotation_virtual_start       = "2024-01-01T00:00:00-05:00"
    rotation_turn_length_seconds = 86400
    users                        = [pagerduty_user.holiday_coverage.id]

    restriction {
      type              = "daily_restriction"
      start_time_of_day = "00:00:00"
      duration_seconds  = 86400
      start_day_of_week = 0  # Sunday
    }
  }
}

yaml

resource "pagerduty_schedule" "primary" {
  name      = "Primary On-Call"
  time_zone = "America/New_York"

  layer {
    name                         = "Weekly Rotation"
    start                        = "2024-01-01T00:00:00-05:00"
    rotation_virtual_start       = "2024-01-01T00:00:00-05:00"
    rotation_turn_length_seconds = 604800  # 1 week
    users                        = [for user in pagerduty_user.oncall : user.id]
  }

  # Override layer for holidays
  layer {
    name                         = "Holiday Coverage"
    start                        = "2024-01-01T00:00:00-05:00"
    rotation_virtual_start       = "2024-01-01T00:00:00-05:00"
    rotation_turn_length_seconds = 86400
    users                        = [pagerduty_user.holiday_coverage.id]

    restriction {
      type              = "daily_restriction"
      start_time_of_day = "00:00:00"
      duration_seconds  = 86400
      start_day_of_week = 0  # Sunday
    }
  }
}

Grafana OnCall

Integration Setup

集成设置

yaml

undefined

yaml

undefined

docker-compose.yml addition

services: oncall: image: grafana/oncall environment: - SECRET_KEY=your-secret-key - BASE_URL=http://oncall:8080 - GRAFANA_API_URL=http://grafana:3000 ports: - "8080:8080"

undefined

services: oncall: image: grafana/oncall environment: - SECRET_KEY=your-secret-key - BASE_URL=http://oncall:8080 - GRAFANA_API_URL=http://grafana:3000 ports: - "8080:8080"

undefined

Escalation Chain

升级链

yaml

undefined

yaml

undefined

Example escalation chain structure

escalation_chains:

name: "Production Critical" steps:
- step: 1 type: notify persons:
  - "@oncall-primary" wait_delay: 0
- step: 2 type: notify persons:
  - "@oncall-secondary" wait_delay: 5m
- step: 3 type: notify persons:
  - "@engineering-manager" wait_delay: 10m
- step: 4 type: trigger_action action: "escalate_to_incident_commander" wait_delay: 15m

undefined

escalation_chains:

name: "Production Critical" steps:
- step: 1 type: notify persons:
  - "@oncall-primary" wait_delay: 0
- step: 2 type: notify persons:
  - "@oncall-secondary" wait_delay: 5m
- step: 3 type: notify persons:
  - "@engineering-manager" wait_delay: 10m
- step: 4 type: trigger_action action: "escalate_to_incident_commander" wait_delay: 15m

undefined

Alert Templates

告警模板

Slack Alert Template

Slack告警模板

{{ define "slack.title" }}
[{{ .Status | toUpper }}{{ if eq .Status "firing" }}:{{ .Alerts.Firing | len }}{{ end }}] {{ .CommonLabels.alertname }}
{{ end }}

{{ define "slack.text" }}
{{ range .Alerts }}
*Alert:* {{ .Annotations.summary }}
*Severity:* {{ .Labels.severity }}
*Description:* {{ .Annotations.description }}
*Runbook:* {{ .Annotations.runbook_url }}
{{ end }}
{{ end }}

{{ define "slack.title" }}
[{{ .Status | toUpper }}{{ if eq .Status "firing" }}:{{ .Alerts.Firing | len }}{{ end }}] {{ .CommonLabels.alertname }}
{{ end }}

{{ define "slack.text" }}
{{ range .Alerts }}
*Alert:* {{ .Annotations.summary }}
*Severity:* {{ .Labels.severity }}
*Description:* {{ .Annotations.description }}
*Runbook:* {{ .Annotations.runbook_url }}
{{ end }}
{{ end }}

PagerDuty Details Template

PagerDuty详情模板

{{ define "pagerduty.firing" }}
{{ range .Alerts.Firing }}
Alert: {{ .Labels.alertname }}
Service: {{ .Labels.service }}
Instance: {{ .Labels.instance }}
Value: {{ .Annotations.value }}
Started: {{ .StartsAt.Format "2006-01-02 15:04:05" }}
{{ end }}
{{ end }}

{{ define "pagerduty.firing" }}
{{ range .Alerts.Firing }}
Alert: {{ .Labels.alertname }}
Service: {{ .Labels.service }}
Instance: {{ .Labels.instance }}
Value: {{ .Annotations.value }}
Started: {{ .StartsAt.Format "2006-01-02 15:04:05" }}
{{ end }}
{{ end }}

On-Call Best Practices

值班最佳实践

Rotation Guidelines

轮换指南

yaml

on_call_guidelines:
  rotation_length: 1 week
  handoff_time: "10:00 AM Monday"
  
  responsibilities:
    - Monitor alerts during shift
    - Respond within SLA (critical: 5min, high: 15min)
    - Document incidents
    - Handoff unresolved issues
    
  support:
    - Secondary on-call for backup
    - Clear escalation path
    - Manager availability for major incidents
    
  wellness:
    - Maximum 1 week on-call per month
    - Comp time after high-alert periods
    - No-interrupt recovery day after shift

yaml

on_call_guidelines:
  rotation_length: 1 week
  handoff_time: "10:00 AM Monday"
  
  responsibilities:
    - Monitor alerts during shift
    - Respond within SLA (critical: 5min, high: 15min)
    - Document incidents
    - Handoff unresolved issues
    
  support:
    - Secondary on-call for backup
    - Clear escalation path
    - Manager availability for major incidents
    
  wellness:
    - Maximum 1 week on-call per month
    - Comp time after high-alert periods
    - No-interrupt recovery day after shift

Runbook Template

运行手册模板

markdown

undefined

markdown

undefined

Alert: High Error Rate

Summary

Error rate has exceeded the threshold of 5% for the service.

Impact

Users may experience errors when accessing the application.

Investigation Steps

Check service logs:
```
kubectl logs -l app=myapp -n production
```

Review recent deployments:

kubectl rollout history deployment/myapp

Check database connectivity:

kubectl exec -it myapp -- nc -zv postgres 5432

Review error traces in APM dashboard

Check service logs:
```
kubectl logs -l app=myapp -n production
```

Review recent deployments:

kubectl rollout history deployment/myapp

Check database connectivity:

kubectl exec -it myapp -- nc -zv postgres 5432

Review error traces in APM dashboard

Remediation

If caused by recent deployment:

bash

kubectl rollout undo deployment/myapp -n production

bash

kubectl rollout undo deployment/myapp -n production

If database related:

bash

kubectl delete pod -l app=postgres -n production

bash

kubectl delete pod -l app=postgres -n production

Escalation

If not resolved within 15 minutes, escalate to:

Database team: @db-oncall
Platform team: @platform-oncall

undefined

If not resolved within 15 minutes, escalate to:

Database team: @db-oncall
Platform team: @platform-oncall

undefined

Alert Fatigue Reduction

告警疲劳缓解

Strategies

策略

yaml

fatigue_reduction:
  aggregate_alerts:
    - Group related alerts
    - Use inhibit rules
    - Implement alert correlation
    
  tune_thresholds:
    - Base on SLOs, not arbitrary values
    - Account for normal variance
    - Use appropriate evaluation windows
    
  automate_responses:
    - Auto-remediation for known issues
    - Self-healing infrastructure
    - Automated scaling
    
  regular_review:
    - Weekly alert review
    - Remove unused alerts
    - Update thresholds based on data

yaml

fatigue_reduction:
  aggregate_alerts:
    - Group related alerts
    - Use inhibit rules
    - Implement alert correlation
    
  tune_thresholds:
    - Base on SLOs, not arbitrary values
    - Account for normal variance
    - Use appropriate evaluation windows
    
  automate_responses:
    - Auto-remediation for known issues
    - Self-healing infrastructure
    - Automated scaling
    
  regular_review:
    - Weekly alert review
    - Remove unused alerts
    - Update thresholds based on data

Common Issues

常见问题

Issue: Alert Storm

问题：告警风暴

Problem: Too many alerts firing simultaneously Solution: Implement proper grouping and inhibition rules

问题：同时触发过多告警 解决方案：实施合理的分组和抑制规则

Issue: Missed Alerts

问题：遗漏告警

Problem: Critical alerts not reaching on-call Solution: Test escalation policies, verify contact methods

问题：关键告警未送达值班人员 解决方案：测试升级策略，验证联系方式

Issue: False Positives

问题：误报

Problem: Alerts firing without actual issues Solution: Tune thresholds, increase evaluation windows

问题：无实际问题却触发告警 解决方案：调整阈值，延长评估窗口

Best Practices

最佳实践

Define clear severity levels
Every alert needs a runbook
Test on-call notifications regularly
Review and tune alerts weekly
Implement proper escalation paths
Use alert grouping and inhibition
Track alert metrics (MTTR, frequency)
Practice incident response regularly

定义清晰的严重级别
每个告警都需配备运行手册
定期测试值班通知
每周审核并调整告警
实施合理的升级路径
使用告警分组和抑制
跟踪告警指标（平均恢复时间、触发频率）
定期演练事件响应

alerting-oncall

Original

Translation

Alerting & On-Call

告警与值班管理

When to Use This Skill

何时使用此技能

Prerequisites

前提条件

Alerting Best Practices

告警最佳实践

Alert Categories

告警分类

Severity levels

Severity levels

Alert Design Principles

告警设计原则

Good alert characteristics

Good alert characteristics

Prometheus Alerting

Prometheus告警配置

Alert Rules

告警规则

prometheus/rules/alerts.yml

prometheus/rules/alerts.yml

High-level service health

Error rate alert

Latency alert (SLO-based)

High-level service health

Error rate alert

Latency alert (SLO-based)

Alertmanager Configuration

Alertmanager配置

alertmanager.yml

alertmanager.yml

PagerDuty Integration

PagerDuty集成

Service Configuration

服务配置

Terraform example

Terraform example

Schedule Configuration

排班配置

Grafana OnCall

Grafana OnCall

Integration Setup

集成设置

docker-compose.yml addition

docker-compose.yml addition

Escalation Chain

升级链

Example escalation chain structure

Example escalation chain structure

Alert Templates

告警模板

Slack Alert Template

Slack告警模板

PagerDuty Details Template

PagerDuty详情模板

On-Call Best Practices

值班最佳实践

Rotation Guidelines

轮换指南

Runbook Template

运行手册模板

Alert: High Error Rate

Alert: High Error Rate

Summary

Summary

Impact

Impact

Investigation Steps

Investigation Steps

Remediation

Remediation

If caused by recent deployment:

If caused by recent deployment:

If database related:

If database related:

Escalation