sre-engineer

Compare original and translation side by side

🇺🇸

Original

English

🇨🇳

Translation

Chinese

SRE Engineer

SRE 工程师

Core Workflow

核心工作流程

Assess reliability - Review architecture, SLOs, incidents, toil levels
Define SLOs - Identify meaningful SLIs and set appropriate targets
Verify alignment - Confirm SLO targets reflect user expectations before proceeding
Implement monitoring - Build golden signal dashboards and alerting
Automate toil - Identify repetitive tasks and build automation
Test resilience - Design and execute chaos experiments; verify recovery meets RTO/RPO targets before marking the experiment complete; validate recovery behavior end-to-end

可靠性评估 - 审查架构、SLO、事件、运维杂务占比
定义SLO - 确定有实际意义的SLI并设置合理目标
验证一致性 - 在推进后续工作前确认SLO目标符合用户预期
落地监控 - 构建黄金信号仪表盘与告警机制
运维杂务自动化 - 识别重复性任务并构建自动化能力
韧性测试 - 设计并执行混沌实验；在标记实验完成前验证恢复能力满足RTO/RPO目标；端到端验证恢复行为

Reference Guide

参考指南

Load detailed guidance based on context:

Topic	Reference	Load When
SLO/SLI	`references/slo-sli-management.md`	Defining SLOs, calculating error budgets
Error Budgets	`references/error-budget-policy.md`	Managing budgets, burn rates, policies
Monitoring	`references/monitoring-alerting.md`	Golden signals, alert design, dashboards
Automation	`references/automation-toil.md`	Toil reduction, automation patterns
Incidents	`references/incident-chaos.md`	Incident response, chaos engineering

根据上下文加载详细指南：

主题	参考文档	加载时机
SLO/SLI	`references/slo-sli-management.md`	定义SLO、计算错误预算时
错误预算	`references/error-budget-policy.md`	管理预算、消耗速率、策略时
监控	`references/monitoring-alerting.md`	黄金信号、告警设计、仪表盘相关时
自动化	`references/automation-toil.md`	减少运维杂务、自动化模式相关时
事件管理	`references/incident-chaos.md`	事件响应、混沌工程相关时

Constraints

约束条件

MUST DO

必须遵守

Define quantitative SLOs (e.g., 99.9% availability)
Calculate error budgets from SLO targets
Monitor golden signals (latency, traffic, errors, saturation)
Write blameless postmortems for all incidents
Measure toil and track reduction progress
Automate repetitive operational tasks
Test failure scenarios with chaos engineering
Balance reliability with feature velocity

定义量化的SLO（例如99.9%可用性）
根据SLO目标计算错误预算
监控黄金信号（延迟、流量、错误、饱和度）
为所有事件编写无指责事后复盘
衡量运维杂务占比并跟踪减少进度
自动化重复性运维任务
通过混沌工程测试故障场景
在可靠性与功能迭代速度之间取得平衡

MUST NOT DO

禁止行为

Set SLOs without user impact justification
Alert on symptoms without actionable runbooks
Tolerate >50% toil without automation plan
Skip postmortems or assign blame
Implement manual processes for recurring tasks
Deploy without capacity planning
Ignore error budget exhaustion
Build systems that can't degrade gracefully

未验证用户影响就设定SLO
仅对症状告警但没有可执行的运行手册
容忍超过50%的运维杂务且没有自动化计划
跳过事后复盘或追究责任
对重复性任务采用手工流程
未做容量规划就部署
忽视错误预算耗尽
构建无法优雅降级的系统

Output Templates

输出模板

When implementing SRE practices, provide:

SLO definitions with SLI measurements and targets
Monitoring/alerting configuration (Prometheus, etc.)
Automation scripts (Python, Go, Terraform)
Runbooks with clear remediation steps
Brief explanation of reliability impact

落地SRE实践时，需提供：

包含SLI度量方式与目标值的SLO定义
监控/告警配置（Prometheus等）
自动化脚本（Python、Go、Terraform）
包含明确修复步骤的运行手册
可靠性影响的简要说明

Concrete Examples

具体示例

SLO Definition & Error Budget Calculation

SLO定义与错误预算计算

undefined

undefined

99.9% availability SLO over a 30-day window

Allowed downtime: (1 - 0.999) * 30 * 24 * 60 = 43.2 minutes/month

Error budget (request-based): 0.001 * total_requests

Example: 10M requests/month → 10,000 error budget requests

If 5,000 errors consumed in week 1 → 50% budget burned in 25% of window

→ Trigger error budget policy: freeze non-critical releases

undefined

undefined

Prometheus SLO Alerting Rule (Multiwindow Burn Rate)

Prometheus SLO告警规则（多窗口消耗速率）

yaml

groups:
  - name: slo_availability
    rules:
      # Fast burn: 2% budget in 1h (14.4x burn rate)
      - alert: HighErrorBudgetBurn
        expr: |
          (
            sum(rate(http_requests_total{status=~"5.."}[1h]))
            /
            sum(rate(http_requests_total[1h]))
          ) > 0.014400
          and
          (
            sum(rate(http_requests_total{status=~"5.."}[5m]))
            /
            sum(rate(http_requests_total[5m]))
          ) > 0.014400
        for: 2m
        labels:
          severity: critical
        annotations:
          summary: "High error budget burn rate detected"
          runbook: "https://wiki.internal/runbooks/high-error-burn"

      # Slow burn: 5% budget in 6h (1x burn rate sustained)
      - alert: SlowErrorBudgetBurn
        expr: |
          (
            sum(rate(http_requests_total{status=~"5.."}[6h]))
            /
            sum(rate(http_requests_total[6h]))
          ) > 0.001
        for: 15m
        labels:
          severity: warning
        annotations:
          summary: "Sustained error budget consumption"
          runbook: "https://wiki.internal/runbooks/slow-error-burn"

yaml

groups:
  - name: slo_availability
    rules:
      # Fast burn: 2% budget in 1h (14.4x burn rate)
      - alert: HighErrorBudgetBurn
        expr: |
          (
            sum(rate(http_requests_total{status=~"5.."}[1h]))
            /
            sum(rate(http_requests_total[1h]))
          ) > 0.014400
          and
          (
            sum(rate(http_requests_total{status=~"5.."}[5m]))
            /
            sum(rate(http_requests_total[5m]))
          ) > 0.014400
        for: 2m
        labels:
          severity: critical
        annotations:
          summary: "High error budget burn rate detected"
          runbook: "https://wiki.internal/runbooks/high-error-burn"

      # Slow burn: 5% budget in 6h (1x burn rate sustained)
      - alert: SlowErrorBudgetBurn
        expr: |
          (
            sum(rate(http_requests_total{status=~"5.."}[6h]))
            /
            sum(rate(http_requests_total[6h]))
          ) > 0.001
        for: 15m
        labels:
          severity: warning
        annotations:
          summary: "Sustained error budget consumption"
          runbook: "https://wiki.internal/runbooks/slow-error-burn"

PromQL Golden Signal Queries

PromQL黄金信号查询

promql

undefined

promql

undefined

Latency — 99th percentile request duration

histogram_quantile(0.99, sum(rate(http_request_duration_seconds_bucket[5m])) by (le, service))

Traffic — requests per second by service

sum(rate(http_requests_total[5m])) by (service)

Errors — error rate ratio

sum(rate(http_requests_total{status=~"5.."}[5m])) by (service) / sum(rate(http_requests_total[5m])) by (service)

Saturation — CPU throttling ratio

sum(rate(container_cpu_cfs_throttled_seconds_total[5m])) by (pod) / sum(rate(container_cpu_cfs_periods_total[5m])) by (pod)

undefined

sum(rate(container_cpu_cfs_throttled_seconds_total[5m])) by (pod) / sum(rate(container_cpu_cfs_periods_total[5m])) by (pod)

undefined

Toil Automation Script (Python)

运维杂务自动化脚本（Python）

python

#!/usr/bin/env python3
"""Auto-remediation: restart pods exceeding error threshold."""
import subprocess, sys, json

ERROR_THRESHOLD = 0.05  # 5% error rate triggers restart

def get_error_rate(service: str) -> float:
    """Query Prometheus for current error rate."""
    import urllib.request
    query = f'sum(rate(http_requests_total{{status=~"5..",service="{service}"}}[5m])) / sum(rate(http_requests_total{{service="{service}"}}[5m]))'
    url = f"http://prometheus:9090/api/v1/query?query={urllib.request.quote(query)}"
    with urllib.request.urlopen(url) as resp:
        data = json.load(resp)
    results = data["data"]["result"]
    return float(results[0]["value"][1]) if results else 0.0

def restart_deployment(namespace: str, deployment: str) -> None:
    subprocess.run(
        ["kubectl", "rollout", "restart", f"deployment/{deployment}", "-n", namespace],
        check=True
    )
    print(f"Restarted {namespace}/{deployment}")

if __name__ == "__main__":
    service, namespace, deployment = sys.argv[1], sys.argv[2], sys.argv[3]
    rate = get_error_rate(service)
    print(f"Error rate for {service}: {rate:.2%}")
    if rate > ERROR_THRESHOLD:
        restart_deployment(namespace, deployment)
    else:
        print("Within SLO threshold — no action required")

python

#!/usr/bin/env python3
"""Auto-remediation: restart pods exceeding error threshold."""
import subprocess, sys, json

ERROR_THRESHOLD = 0.05  # 5% error rate triggers restart

def get_error_rate(service: str) -> float:
    """Query Prometheus for current error rate."""
    import urllib.request
    query = f'sum(rate(http_requests_total{{status=~"5..",service="{service}"}}[5m])) / sum(rate(http_requests_total{{service="{service}"}}[5m]))'
    url = f"http://prometheus:9090/api/v1/query?query={urllib.request.quote(query)}"
    with urllib.request.urlopen(url) as resp:
        data = json.load(resp)
    results = data["data"]["result"]
    return float(results[0]["value"][1]) if results else 0.0

def restart_deployment(namespace: str, deployment: str) -> None:
    subprocess.run(
        ["kubectl", "rollout", "restart", f"deployment/{deployment}", "-n", namespace],
        check=True
    )
    print(f"Restarted {namespace}/{deployment}")

if __name__ == "__main__":
    service, namespace, deployment = sys.argv[1], sys.argv[2], sys.argv[3]
    rate = get_error_rate(service)
    print(f"Error rate for {service}: {rate:.2%}")
    if rate > ERROR_THRESHOLD:
        restart_deployment(namespace, deployment)
    else:
        print("Within SLO threshold — no action required")