Back to Details

monitoring-expert

Compare original and translation side by side

🇺🇸

Original

English
🇨🇳

Translation

Chinese

Monitoring Expert

监控专家

Observability and performance specialist implementing comprehensive monitoring, alerting, tracing, and performance testing systems.
可观测性与性能专家，负责搭建完善的监控、告警、链路追踪和性能测试体系。

Core Workflow

核心工作流

  1. Assess — Identify what needs monitoring (SLIs, critical paths, business metrics)
  2. Instrument — Add logging, metrics, and traces to the application (see examples below)
  3. Collect — Configure aggregation and storage (Prometheus scrape, log shipper, OTLP endpoint); verify data arrives before proceeding
  4. Visualize — Build dashboards using RED (Rate/Errors/Duration) or USE (Utilization/Saturation/Errors) methods
  5. Alert — Define threshold and anomaly alerts on critical paths; validate no false-positive flood before shipping
  1. 评估 — 确定需要监控的内容（SLI、关键路径、业务指标）
  2. 埋点 — 为应用添加日志、指标和链路追踪（见下方示例）
  3. 采集 — 配置聚合与存储（Prometheus抓取、日志采集器、OTLP端点）；确认数据正常上报后再继续下一步
  4. 可视化 — 使用RED（速率/错误/耗时）或USE（利用率/饱和度/错误）方法搭建仪表盘
  5. 告警 — 为关键路径定义阈值告警和异常告警；上线前验证无误报泛滥问题

Quick-Start Examples

快速上手示例

Structured Logging (Node.js / Pino)

结构化日志（Node.js / Pino）

js
import pino from 'pino';

const logger = pino({ level: 'info' });

// Good — structured fields, includes correlation ID
logger.info({ requestId: req.id, userId: req.user.id, durationMs: elapsed }, 'order.created');

// Bad — string interpolation, no correlation
console.log(`Order created for user ${userId}`);
js
import pino from 'pino';

const logger = pino({ level: 'info' });

// Good — structured fields, includes correlation ID
logger.info({ requestId: req.id, userId: req.user.id, durationMs: elapsed }, 'order.created');

// Bad — string interpolation, no correlation
console.log(`Order created for user ${userId}`);

Prometheus Metrics (Node.js)

Prometheus指标（Node.js）

js
import { Counter, Histogram, register } from 'prom-client';

const httpRequests = new Counter({
  name: 'http_requests_total',
  help: 'Total HTTP requests',
  labelNames: ['method', 'route', 'status'],
});

const httpDuration = new Histogram({
  name: 'http_request_duration_seconds',
  help: 'HTTP request latency',
  labelNames: ['method', 'route'],
  buckets: [0.05, 0.1, 0.3, 0.5, 1, 2, 5],
});

// Instrument a route
app.use((req, res, next) => {
  const end = httpDuration.startTimer({ method: req.method, route: req.path });
  res.on('finish', () => {
    httpRequests.inc({ method: req.method, route: req.path, status: res.statusCode });
    end();
  });
  next();
});

// Expose scrape endpoint
app.get('/metrics', async (req, res) => {
  res.set('Content-Type', register.contentType);
  res.end(await register.metrics());
});
js
import { Counter, Histogram, register } from 'prom-client';

const httpRequests = new Counter({
  name: 'http_requests_total',
  help: 'Total HTTP requests',
  labelNames: ['method', 'route', 'status'],
});

const httpDuration = new Histogram({
  name: 'http_request_duration_seconds',
  help: 'HTTP request latency',
  labelNames: ['method', 'route'],
  buckets: [0.05, 0.1, 0.3, 0.5, 1, 2, 5],
});

// Instrument a route
app.use((req, res, next) => {
  const end = httpDuration.startTimer({ method: req.method, route: req.path });
  res.on('finish', () => {
    httpRequests.inc({ method: req.method, route: req.path, status: res.statusCode });
    end();
  });
  next();
});

// Expose scrape endpoint
app.get('/metrics', async (req, res) => {
  res.set('Content-Type', register.contentType);
  res.end(await register.metrics());
});

OpenTelemetry Tracing (Node.js)

OpenTelemetry链路追踪（Node.js）

js
import { NodeSDK } from '@opentelemetry/sdk-node';
import { OTLPTraceExporter } from '@opentelemetry/exporter-trace-otlp-http';
import { trace } from '@opentelemetry/api';

const sdk = new NodeSDK({
  traceExporter: new OTLPTraceExporter({ url: 'http://jaeger:4318/v1/traces' }),
});
sdk.start();

// Manual span around a critical operation
const tracer = trace.getTracer('order-service');
async function processOrder(orderId) {
  const span = tracer.startSpan('order.process');
  span.setAttribute('order.id', orderId);
  try {
    const result = await db.saveOrder(orderId);
    span.setStatus({ code: SpanStatusCode.OK });
    return result;
  } catch (err) {
    span.recordException(err);
    span.setStatus({ code: SpanStatusCode.ERROR });
    throw err;
  } finally {
    span.end();
  }
}
js
import { NodeSDK } from '@opentelemetry/sdk-node';
import { OTLPTraceExporter } from '@opentelemetry/exporter-trace-otlp-http';
import { trace } from '@opentelemetry/api';

const sdk = new NodeSDK({
  traceExporter: new OTLPTraceExporter({ url: 'http://jaeger:4318/v1/traces' }),
});
sdk.start();

// Manual span around a critical operation
const tracer = trace.getTracer('order-service');
async function processOrder(orderId) {
  const span = tracer.startSpan('order.process');
  span.setAttribute('order.id', orderId);
  try {
    const result = await db.saveOrder(orderId);
    span.setStatus({ code: SpanStatusCode.OK });
    return result;
  } catch (err) {
    span.recordException(err);
    span.setStatus({ code: SpanStatusCode.ERROR });
    throw err;
  } finally {
    span.end();
  }
}

Prometheus Alerting Rule

Prometheus告警规则

yaml
groups:
  - name: api.rules
    rules:
      - alert: HighErrorRate
        expr: |
          rate(http_requests_total{status=~"5.."}[5m])
          / rate(http_requests_total[5m]) > 0.05
        for: 2m
        labels:
          severity: critical
        annotations:
          summary: "Error rate above 5% on {{ $labels.route }}"
yaml
groups:
  - name: api.rules
    rules:
      - alert: HighErrorRate
        expr: |
          rate(http_requests_total{status=~"5.."}[5m])
          / rate(http_requests_total[5m]) > 0.05
        for: 2m
        labels:
          severity: critical
        annotations:
          summary: "Error rate above 5% on {{ $labels.route }}"

k6 Load Test

k6负载测试

js
import http from 'k6/http';
import { check, sleep } from 'k6';

export const options = {
  stages: [
    { duration: '1m', target: 50 },   // ramp up
    { duration: '5m', target: 50 },   // sustained load
    { duration: '1m', target: 0 },    // ramp down
  ],
  thresholds: {
    http_req_duration: ['p(95)<500'],  // 95th percentile < 500 ms
    http_req_failed:   ['rate<0.01'],  // error rate < 1%
  },
};

export default function () {
  const res = http.get('https://api.example.com/orders');
  check(res, { 'status is 200': (r) => r.status === 200 });
  sleep(1);
}
js
import http from 'k6/http';
import { check, sleep } from 'k6';

export const options = {
  stages: [
    { duration: '1m', target: 50 },   // ramp up
    { duration: '5m', target: 50 },   // sustained load
    { duration: '1m', target: 0 },    // ramp down
  ],
  thresholds: {
    http_req_duration: ['p(95)<500'],  // 95th percentile < 500 ms
    http_req_failed:   ['rate<0.01'],  // error rate < 1%
  },
};

export default function () {
  const res = http.get('https://api.example.com/orders');
  check(res, { 'status is 200': (r) => r.status === 200 });
  sleep(1);
}

Reference Guide

参考指南

Load detailed guidance based on context:
TopicReferenceLoad When
Logging
references/structured-logging.md
Pino, JSON logging
Metrics
references/prometheus-metrics.md
Counter, Histogram, Gauge
Tracing
references/opentelemetry.md
OpenTelemetry, spans
Alerting
references/alerting-rules.md
Prometheus alerts
Dashboards
references/dashboards.md
RED/USE method, Grafana
Performance Testing
references/performance-testing.md
Load testing, k6, Artillery, benchmarks
Profiling
references/application-profiling.md
CPU/memory profiling, bottlenecks
Capacity Planning
references/capacity-planning.md
Scaling, forecasting, budgets
可根据场景加载详细指南：
主题参考文件触发加载场景
日志
references/structured-logging.md
Pino、JSON日志
指标
references/prometheus-metrics.md
Counter、Histogram、Gauge
链路追踪
references/opentelemetry.md
OpenTelemetry、Span
告警
references/alerting-rules.md
Prometheus告警
仪表盘
references/dashboards.md
RED/USE方法、Grafana
性能测试
references/performance-testing.md
负载测试、k6、Artillery、基准测试
性能剖析
references/application-profiling.md
CPU/内存剖析、瓶颈分析
容量规划
references/capacity-planning.md
扩缩容、容量预测、预算规划

Constraints

约束规范

MUST DO

必须遵守

  • Use structured logging (JSON)
  • Include request IDs for correlation
  • Set up alerts for critical paths
  • Monitor business metrics, not just technical
  • Use appropriate metric types (counter/gauge/histogram)
  • Implement health check endpoints
  • 使用结构化日志（JSON格式）
  • 包含请求ID用于链路关联
  • 为关键路径配置告警
  • 不仅要监控技术指标，也要监控业务指标
  • 使用合适的指标类型（counter/gauge/histogram）
  • 实现健康检查端点

MUST NOT DO

禁止行为

  • Log sensitive data (passwords, tokens, PII)
  • Alert on every error (alert fatigue)
  • Use string interpolation in logs (use structured fields)
  • Skip correlation IDs in distributed systems
  • 记录敏感数据（密码、令牌、PII）
  • 对每个错误都触发告警（导致告警疲劳）
  • 在日志中使用字符串插值（应使用结构化字段）
  • 在分布式系统中省略关联ID