Back to Details

sre-dashboards

Compare original and translation side by side

🇺🇸

Original

English
🇨🇳

Translation

Chinese

SRE Dashboards

SRE仪表盘

Build dashboards that help teams detect, triage, and prevent reliability incidents.
构建可帮助团队检测、分类和预防可靠性事件的仪表盘。

When to Use This Skill

何时使用该技能

Use this skill when:
  • Defining service-level dashboards for production systems
  • Tracking SLO health and error-budget burn
  • Creating incident command-center views
  • Standardizing dashboard patterns across teams
在以下场景使用本技能：
  • 为生产系统定义服务级仪表盘
  • 跟踪SLO健康状况和错误预算消耗
  • 创建事件指挥中心视图
  • 在团队间标准化仪表盘模式

Prerequisites

前置条件

  • Metrics pipeline (Prometheus, OpenTelemetry, or vendor equivalent)
  • Logs/traces linked to services and environments
  • Agreed service taxonomy (team, service, tier, environment)
  • 指标流水线（Prometheus、OpenTelemetry或同类厂商工具）
  • 与服务和环境关联的日志/追踪数据
  • 已达成共识的服务分类体系（团队、服务、层级、环境）

Dashboard Architecture

仪表盘架构

Structure dashboards in layers:
  1. Executive Reliability View: SLO attainment, incident counts, MTTR trends.
  2. Service Health View: RED/USE metrics, dependency health, release markers.
  3. Deep-Dive View: Per-endpoint latency, resource saturation, error categories.
Keep each view answer-oriented:
  • Are customers impacted?
  • What changed?
  • Where is the bottleneck?
按层级构建仪表盘：
  1. 高层可靠性视图：SLO达成率、事件数量、平均恢复时间趋势。
  2. 服务健康视图：RED/USE指标、依赖健康状况、发布标记。
  3. 深度排查视图：每个端点的延迟、资源饱和度、错误类别。
确保每个视图都聚焦于解决以下问题：
  • 客户是否受到影响？
  • 发生了哪些变化？
  • 瓶颈在哪里？

Core SRE Panels

核心SRE面板

Golden Signals

黄金指标

  • Latency: p50/p95/p99 request duration by endpoint
  • Traffic: request throughput and queue depth
  • Errors: 5xx rate, failed jobs, timeout ratio
  • Saturation: CPU, memory, disk I/O, thread/connection pool exhaustion
  • 延迟：按端点统计的p50/p95/p99请求时长
  • 流量：请求吞吐量和队列深度
  • 错误：5xx错误率、失败任务数、超时比例
  • 饱和度：CPU、内存、磁盘I/O、线程/连接池耗尽情况

SLO Panels

SLO面板

  • Current SLI value (rolling windows: 5m, 1h, 24h, 30d)
  • Error-budget remaining (%)
  • Burn-rate panels (fast and slow windows)
  • Multi-window burn alert status
  • 当前SLI值（滚动窗口：5分钟、1小时、24小时、30天）
  • 剩余错误预算（百分比）
  • 消耗速率面板（快速和慢速窗口）
  • 多窗口消耗告警状态

Change Correlation

变更关联

  • Deployment markers and config-change annotations
  • Feature flag state overlays
  • Upstream/downstream dependency error rates
  • 部署标记和配置变更注释
  • 功能标志状态叠加层
  • 上游/下游依赖错误率

Example PromQL Snippets

PromQL示例代码片段

promql
undefined
promql
undefined

API error rate (%)

API error rate (%)

100 * sum(rate(http_requests_total{status=~"5.."}[5m])) / sum(rate(http_requests_total[5m]))

```promql
100 * sum(rate(http_requests_total{status=~"5.."}[5m])) / sum(rate(http_requests_total[5m]))

```promql

p95 latency by route

p95 latency by route

histogram_quantile(0.95, sum by (le, route) (rate(http_request_duration_seconds_bucket[5m])) )

```promql
histogram_quantile(0.95, sum by (le, route) (rate(http_request_duration_seconds_bucket[5m])) )

```promql

Fast burn rate (5m / 1h)

Fast burn rate (5m / 1h)

( sum(rate(http_requests_total{status="5.."}[5m])) / sum(rate(http_requests_total[5m])) ) / ( sum(rate(http_requests_total{status="5.."}[1h])) / sum(rate(http_requests_total[1h])) )
undefined
( sum(rate(http_requests_total{status="5.."}[5m])) / sum(rate(http_requests_total[5m])) ) / ( sum(rate(http_requests_total{status="5.."}[1h])) / sum(rate(http_requests_total[1h])) )
undefined

Operational Guidelines

运维指南

  • Use consistent color semantics (green=healthy, yellow=degrading, red=breach)
  • Label units explicitly (ms, req/s, %, cores)
  • Default time windows to incident-friendly ranges (15m, 1h, 6h, 24h)
  • Minimize panel count per dashboard to reduce cognitive load
  • Add runbook links directly in panel descriptions
  • 使用一致的颜色语义（绿色=健康，黄色=降级，红色=违规）
  • 明确标注单位（毫秒、请求/秒、百分比、核心数）
  • 默认时间窗口设置为适合事件处理的范围（15分钟、1小时、6小时、24小时）
  • 减少每个仪表盘的面板数量以降低认知负荷
  • 在面板描述中直接添加运行手册链接

Troubleshooting

故障排查

Panel appears flat or empty

面板显示平坦或为空

  • Verify label cardinality and filters (
    service
    , 
    env
    , 
    region
    )
  • Confirm scrape/ingest latency is within expected range
  • Check metric rename regressions after instrumentation updates
  • 验证标签基数和过滤器（
    service
    、
    env
    、
    region
    ）
  • 确认采集/摄入延迟在预期范围内
  • 检查 instrumentation 更新后的指标命名回归问题

High cardinality slows dashboards

高基数导致仪表盘运行缓慢

  • Aggregate by stable dimensions (
    service
    , 
    route_group
    ) instead of raw IDs
  • Use recording rules for expensive percentile and ratio queries
  • Split deep-dive dashboards from NOC summary dashboards
  • 按稳定维度（
    service
    、
    route_group
    ）聚合，而非原始ID
  • 对昂贵的百分位数和比率查询使用记录规则
  • 将深度排查仪表盘与NOC汇总仪表盘分离

Related Skills

相关技能

  • prometheus-grafana - Dashboard implementation and PromQL
  • opentelemetry - Standardized telemetry instrumentation
  • alerting-oncall - Reliability alert routing and escalation
  • agent-observability - AI workload reliability telemetry
  • prometheus-grafana - 仪表盘实现与PromQL
  • opentelemetry - 标准化遥测 instrumentation
  • alerting-oncall - 可靠性告警路由与升级
  • agent-observability - AI工作负载可靠性遥测