opentelemetry
Compare original and translation side by side
🇺🇸
Original
English🇨🇳
Translation
ChineseOpenTelemetry
OpenTelemetry
Adopt vendor-neutral telemetry with consistent instrumentation across services.
采用厂商中立的遥测方案,在所有服务中实现统一的可观测性埋点。
When to Use This Skill
适用场景
- Debugging latency across microservices
- Standardizing observability data model and naming
- Sending telemetry to Prometheus, Grafana, Datadog, or OTLP backends
- Building SLO dashboards with trace-to-log correlation
- Instrumenting Python or Node.js applications with tracing and metrics
- Setting up auto-instrumentation for existing services without code changes
- 排查微服务间的延迟问题
- 标准化可观测性数据模型与命名规范
- 将遥测数据发送至Prometheus、Grafana、Datadog或OTLP后端
- 构建支持链路-日志关联的SLO仪表盘
- 为Python或Node.js应用添加链路追踪与指标埋点
- 为现有服务设置自动埋点,无需修改代码
Prerequisites
前置条件
- Application services running in containers or on VMs
- Backend for traces (Jaeger, Tempo, Datadog, or any OTLP receiver)
- Backend for metrics (Prometheus, Mimir, or OTLP receiver)
- Kubernetes cluster (for collector deployment) or VM with systemd
- Network access from services to collector, and collector to backends
- 运行在容器或VM中的应用服务
- 链路追踪后端(Jaeger、Tempo、Datadog或任意OTLP接收器)
- 指标采集后端(Prometheus、Mimir或OTLP接收器)
- Kubernetes集群(用于Collector部署)或带有systemd的VM
- 服务到Collector、Collector到后端的网络访问权限
Core Workflow
核心工作流程
- Define semantic conventions for services, environments, and versions.
- Add SDK or auto-instrumentation in each service.
- Run an OpenTelemetry Collector to receive, transform, and export telemetry.
- Validate cardinality and sampling to control cost.
- Create golden signals dashboards and alerting from collected data.
- 定义服务、环境和版本的语义规范
- 在每个服务中添加SDK或启用自动埋点
- 运行OpenTelemetry Collector以接收、转换并导出遥测数据
- 验证基数与采样策略,控制成本
- 基于采集的数据创建黄金信号仪表盘与告警规则
Collector Production Configuration
生产环境Collector配置
yaml
undefinedyaml
undefinedotel-collector-config.yaml
otel-collector-config.yaml
receivers:
otlp:
protocols:
grpc:
endpoint: 0.0.0.0:4317
http:
endpoint: 0.0.0.0:4318
Scrape Prometheus endpoints
prometheus:
config:
scrape_configs:
- job_name: "kubernetes-pods"
kubernetes_sd_configs:
- role: pod
relabel_configs:
- source_labels: [__meta_kubernetes_pod_annotation_prometheus_io_scrape]
action: keep
regex: "true"
- source_labels: [__meta_kubernetes_pod_annotation_prometheus_io_port]
action: replace
target_label: address
regex: (.+)
replacement: $$1
Host metrics for infrastructure monitoring
hostmetrics:
collection_interval: 30s
scrapers:
cpu: {}
memory: {}
disk: {}
network: {}
load: {}
processors:
batch:
send_batch_size: 1024
timeout: 5s
memory_limiter:
check_interval: 1s
limit_mib: 512
spike_limit_mib: 128
attributes:
actions:
- key: deployment.environment
value: production
action: upsert
Drop high-cardinality attributes to control cost
filter/drop-debug:
traces:
span:
- 'attributes["http.request.header.x-debug"] == "true"'
Reduce cardinality on URL paths
transform/normalize-routes:
trace_statements:
- context: span
statements:
- replace_pattern(attributes["url.path"], "/users/[0-9]+", "/users/{id}")
- replace_pattern(attributes["url.path"], "/orders/[0-9]+", "/orders/{id}")
Resource detection for cloud environments
resourcedetection:
detectors: [env, system, gcp, aws, azure]
timeout: 5s
exporters:
Send traces to Tempo/Jaeger
otlp/traces:
endpoint: tempo:4317
tls:
insecure: true
Send metrics to Prometheus via remote write
prometheusremotewrite:
endpoint: http://mimir:9009/api/v1/push
tls:
insecure: true
Send logs to Loki
otlp/logs:
endpoint: loki:4317
tls:
insecure: true
Debug exporter for development
debug:
verbosity: basic
service:
telemetry:
logs:
level: info
metrics:
address: 0.0.0.0:8888
pipelines:
traces:
receivers: [otlp]
processors: [memory_limiter, resourcedetection, transform/normalize-routes, batch, attributes]
exporters: [otlp/traces]
metrics:
receivers: [otlp, prometheus, hostmetrics]
processors: [memory_limiter, resourcedetection, batch, attributes]
exporters: [prometheusremotewrite]
logs:
receivers: [otlp]
processors: [memory_limiter, resourcedetection, batch, attributes]
exporters: [otlp/logs]
undefinedreceivers:
otlp:
protocols:
grpc:
endpoint: 0.0.0.0:4317
http:
endpoint: 0.0.0.0:4318
Scrape Prometheus endpoints
prometheus:
config:
scrape_configs:
- job_name: "kubernetes-pods"
kubernetes_sd_configs:
- role: pod
relabel_configs:
- source_labels: [__meta_kubernetes_pod_annotation_prometheus_io_scrape]
action: keep
regex: "true"
- source_labels: [__meta_kubernetes_pod_annotation_prometheus_io_port]
action: replace
target_label: address
regex: (.+)
replacement: $$1
Host metrics for infrastructure monitoring
hostmetrics:
collection_interval: 30s
scrapers:
cpu: {}
memory: {}
disk: {}
network: {}
load: {}
processors:
batch:
send_batch_size: 1024
timeout: 5s
memory_limiter:
check_interval: 1s
limit_mib: 512
spike_limit_mib: 128
attributes:
actions:
- key: deployment.environment
value: production
action: upsert
Drop high-cardinality attributes to control cost
filter/drop-debug:
traces:
span:
- 'attributes["http.request.header.x-debug"] == "true"'
Reduce cardinality on URL paths
transform/normalize-routes:
trace_statements:
- context: span
statements:
- replace_pattern(attributes["url.path"], "/users/[0-9]+", "/users/{id}")
- replace_pattern(attributes["url.path"], "/orders/[0-9]+", "/orders/{id}")
Resource detection for cloud environments
resourcedetection:
detectors: [env, system, gcp, aws, azure]
timeout: 5s
exporters:
Send traces to Tempo/Jaeger
otlp/traces:
endpoint: tempo:4317
tls:
insecure: true
Send metrics to Prometheus via remote write
prometheusremotewrite:
endpoint: http://mimir:9009/api/v1/push
tls:
insecure: true
Send logs to Loki
otlp/logs:
endpoint: loki:4317
tls:
insecure: true
Debug exporter for development
debug:
verbosity: basic
service:
telemetry:
logs:
level: info
metrics:
address: 0.0.0.0:8888
pipelines:
traces:
receivers: [otlp]
processors: [memory_limiter, resourcedetection, transform/normalize-routes, batch, attributes]
exporters: [otlp/traces]
metrics:
receivers: [otlp, prometheus, hostmetrics]
processors: [memory_limiter, resourcedetection, batch, attributes]
exporters: [prometheusremotewrite]
logs:
receivers: [otlp]
processors: [memory_limiter, resourcedetection, batch, attributes]
exporters: [otlp/logs]
undefinedCollector Kubernetes Deployment
Kubernetes环境Collector部署
yaml
undefinedyaml
undefinedotel-collector-deployment.yaml
otel-collector-deployment.yaml
apiVersion: apps/v1 kind: Deployment metadata: name: otel-collector namespace: observability spec: replicas: 2 selector: matchLabels: app: otel-collector template: metadata: labels: app: otel-collector spec: containers: - name: collector image: otel/opentelemetry-collector-contrib:0.98.0 args: ["--config=/etc/otel/config.yaml"] ports: - containerPort: 4317 name: otlp-grpc - containerPort: 4318 name: otlp-http - containerPort: 8888 name: metrics resources: requests: cpu: 200m memory: 256Mi limits: cpu: "1" memory: 512Mi volumeMounts: - name: config mountPath: /etc/otel livenessProbe: httpGet: path: / port: 13133 readinessProbe: httpGet: path: / port: 13133 volumes: - name: config configMap: name: otel-collector-config
apiVersion: v1
kind: Service
metadata:
name: otel-collector
namespace: observability
spec:
selector:
app: otel-collector
ports:
- name: otlp-grpc
port: 4317
targetPort: 4317
- name: otlp-http
port: 4318
targetPort: 4318
- name: metrics
port: 8888
targetPort: 8888
undefinedapiVersion: apps/v1 kind: Deployment metadata: name: otel-collector namespace: observability spec: replicas: 2 selector: matchLabels: app: otel-collector template: metadata: labels: app: otel-collector spec: containers: - name: collector image: otel/opentelemetry-collector-contrib:0.98.0 args: ["--config=/etc/otel/config.yaml"] ports: - containerPort: 4317 name: otlp-grpc - containerPort: 4318 name: otlp-http - containerPort: 8888 name: metrics resources: requests: cpu: 200m memory: 256Mi limits: cpu: "1" memory: 512Mi volumeMounts: - name: config mountPath: /etc/otel livenessProbe: httpGet: path: / port: 13133 readinessProbe: httpGet: path: / port: 13133 volumes: - name: config configMap: name: otel-collector-config
apiVersion: v1
kind: Service
metadata:
name: otel-collector
namespace: observability
spec:
selector:
app: otel-collector
ports:
- name: otlp-grpc
port: 4317
targetPort: 4317
- name: otlp-http
port: 4318
targetPort: 4318
- name: metrics
port: 8888
targetPort: 8888
undefinedPython SDK Instrumentation
Python SDK埋点实现
python
undefinedpython
undefinedtracing_setup.py
tracing_setup.py
"""Initialize OpenTelemetry tracing and metrics for a Python service."""
from opentelemetry import trace, metrics
from opentelemetry.sdk.trace import TracerProvider
from opentelemetry.sdk.trace.export import BatchSpanProcessor
from opentelemetry.sdk.metrics import MeterProvider
from opentelemetry.sdk.metrics.export import PeriodicExportingMetricReader
from opentelemetry.sdk.resources import Resource
from opentelemetry.exporter.otlp.proto.grpc.trace_exporter import OTLPSpanExporter
from opentelemetry.exporter.otlp.proto.grpc.metric_exporter import OTLPMetricExporter
from opentelemetry.instrumentation.requests import RequestsInstrumentor
from opentelemetry.instrumentation.flask import FlaskInstrumentor
from opentelemetry.instrumentation.sqlalchemy import SQLAlchemyInstrumentor
import os
def init_telemetry(service_name: str, service_version: str):
"""Initialize OTel SDK with traces and metrics."""
resource = Resource.create({
"service.name": service_name,
"service.version": service_version,
"deployment.environment": os.getenv("DEPLOY_ENV", "development"),
})
# Traces
trace_exporter = OTLPSpanExporter(
endpoint=os.getenv("OTEL_EXPORTER_OTLP_ENDPOINT", "http://otel-collector:4317"),
insecure=True,
)
tracer_provider = TracerProvider(resource=resource)
tracer_provider.add_span_processor(BatchSpanProcessor(trace_exporter))
trace.set_tracer_provider(tracer_provider)
# Metrics
metric_exporter = OTLPMetricExporter(
endpoint=os.getenv("OTEL_EXPORTER_OTLP_ENDPOINT", "http://otel-collector:4317"),
insecure=True,
)
metric_reader = PeriodicExportingMetricReader(metric_exporter, export_interval_millis=15000)
meter_provider = MeterProvider(resource=resource, metric_readers=[metric_reader])
metrics.set_meter_provider(meter_provider)
# Auto-instrument common libraries
RequestsInstrumentor().instrument()
SQLAlchemyInstrumentor().instrument()
return trace.get_tracer(service_name), metrics.get_meter(service_name)"""Initialize OpenTelemetry tracing and metrics for a Python service."""
from opentelemetry import trace, metrics
from opentelemetry.sdk.trace import TracerProvider
from opentelemetry.sdk.trace.export import BatchSpanProcessor
from opentelemetry.sdk.metrics import MeterProvider
from opentelemetry.sdk.metrics.export import PeriodicExportingMetricReader
from opentelemetry.sdk.resources import Resource
from opentelemetry.exporter.otlp.proto.grpc.trace_exporter import OTLPSpanExporter
from opentelemetry.exporter.otlp.proto.grpc.metric_exporter import OTLPMetricExporter
from opentelemetry.instrumentation.requests import RequestsInstrumentor
from opentelemetry.instrumentation.flask import FlaskInstrumentor
from opentelemetry.instrumentation.sqlalchemy import SQLAlchemyInstrumentor
import os
def init_telemetry(service_name: str, service_version: str):
"""Initialize OTel SDK with traces and metrics."""
resource = Resource.create({
"service.name": service_name,
"service.version": service_version,
"deployment.environment": os.getenv("DEPLOY_ENV", "development"),
})
# Traces
trace_exporter = OTLPSpanExporter(
endpoint=os.getenv("OTEL_EXPORTER_OTLP_ENDPOINT", "http://otel-collector:4317"),
insecure=True,
)
tracer_provider = TracerProvider(resource=resource)
tracer_provider.add_span_processor(BatchSpanProcessor(trace_exporter))
trace.set_tracer_provider(tracer_provider)
# Metrics
metric_exporter = OTLPMetricExporter(
endpoint=os.getenv("OTEL_EXPORTER_OTLP_ENDPOINT", "http://otel-collector:4317"),
insecure=True,
)
metric_reader = PeriodicExportingMetricReader(metric_exporter, export_interval_millis=15000)
meter_provider = MeterProvider(resource=resource, metric_readers=[metric_reader])
metrics.set_meter_provider(meter_provider)
# Auto-instrument common libraries
RequestsInstrumentor().instrument()
SQLAlchemyInstrumentor().instrument()
return trace.get_tracer(service_name), metrics.get_meter(service_name)Usage example
Usage example
tracer, meter = init_telemetry("order-service", "1.2.0")
tracer, meter = init_telemetry("order-service", "1.2.0")
Custom span
Custom span
with tracer.start_as_current_span("process_order") as span:
span.set_attribute("order.id", order_id)
span.set_attribute("order.total", total)
# ... business logic ...
with tracer.start_as_current_span("process_order") as span:
span.set_attribute("order.id", order_id)
span.set_attribute("order.total", total)
# ... business logic ...
Custom metric
Custom metric
request_counter = meter.create_counter(
"app.requests",
description="Total application requests",
)
request_counter.add(1, {"route": "/api/orders", "method": "POST"})
undefinedrequest_counter = meter.create_counter(
"app.requests",
description="Total application requests",
)
request_counter.add(1, {"route": "/api/orders", "method": "POST"})
undefinedNode.js SDK Instrumentation
Node.js SDK埋点实现
javascript
// tracing.js
// Initialize OpenTelemetry for a Node.js service.
// Load this file BEFORE any other imports: node -r ./tracing.js app.js
const { NodeSDK } = require("@opentelemetry/sdk-node");
const { OTLPTraceExporter } = require("@opentelemetry/exporter-trace-otlp-grpc");
const { OTLPMetricExporter } = require("@opentelemetry/exporter-metrics-otlp-grpc");
const { PeriodicExportingMetricReader } = require("@opentelemetry/sdk-metrics");
const { getNodeAutoInstrumentations } = require("@opentelemetry/auto-instrumentations-node");
const { Resource } = require("@opentelemetry/resources");
const { ATTR_SERVICE_NAME, ATTR_SERVICE_VERSION } = require("@opentelemetry/semantic-conventions");
const resource = new Resource({
[ATTR_SERVICE_NAME]: process.env.SERVICE_NAME || "node-service",
[ATTR_SERVICE_VERSION]: process.env.SERVICE_VERSION || "1.0.0",
"deployment.environment": process.env.DEPLOY_ENV || "development",
});
const sdk = new NodeSDK({
resource,
traceExporter: new OTLPTraceExporter({
url: process.env.OTEL_EXPORTER_OTLP_ENDPOINT || "http://otel-collector:4317",
}),
metricReader: new PeriodicExportingMetricReader({
exporter: new OTLPMetricExporter({
url: process.env.OTEL_EXPORTER_OTLP_ENDPOINT || "http://otel-collector:4317",
}),
exportIntervalMillis: 15000,
}),
instrumentations: [
getNodeAutoInstrumentations({
"@opentelemetry/instrumentation-http": {
ignoreIncomingPaths: ["/health", "/ready"],
},
"@opentelemetry/instrumentation-express": { enabled: true },
"@opentelemetry/instrumentation-pg": { enabled: true },
"@opentelemetry/instrumentation-redis": { enabled: true },
}),
],
});
sdk.start();
process.on("SIGTERM", () => sdk.shutdown());javascript
// tracing.js
// Initialize OpenTelemetry for a Node.js service.
// Load this file BEFORE any other imports: node -r ./tracing.js app.js
const { NodeSDK } = require("@opentelemetry/sdk-node");
const { OTLPTraceExporter } = require("@opentelemetry/exporter-trace-otlp-grpc");
const { OTLPMetricExporter } = require("@opentelemetry/exporter-metrics-otlp-grpc");
const { PeriodicExportingMetricReader } = require("@opentelemetry/sdk-metrics");
const { getNodeAutoInstrumentations } = require("@opentelemetry/auto-instrumentations-node");
const { Resource } = require("@opentelemetry/resources");
const { ATTR_SERVICE_NAME, ATTR_SERVICE_VERSION } = require("@opentelemetry/semantic-conventions");
const resource = new Resource({
[ATTR_SERVICE_NAME]: process.env.SERVICE_NAME || "node-service",
[ATTR_SERVICE_VERSION]: process.env.SERVICE_VERSION || "1.0.0",
"deployment.environment": process.env.DEPLOY_ENV || "development",
});
const sdk = new NodeSDK({
resource,
traceExporter: new OTLPTraceExporter({
url: process.env.OTEL_EXPORTER_OTLP_ENDPOINT || "http://otel-collector:4317",
}),
metricReader: new PeriodicExportingMetricReader({
exporter: new OTLPMetricExporter({
url: process.env.OTEL_EXPORTER_OTLP_ENDPOINT || "http://otel-collector:4317",
}),
exportIntervalMillis: 15000,
}),
instrumentations: [
getNodeAutoInstrumentations({
"@opentelemetry/instrumentation-http": {
ignoreIncomingPaths: ["/health", "/ready"],
},
"@opentelemetry/instrumentation-express": { enabled: true },
"@opentelemetry/instrumentation-pg": { enabled: true },
"@opentelemetry/instrumentation-redis": { enabled: true },
}),
],
});
sdk.start();
process.on("SIGTERM", () => sdk.shutdown());Auto-Instrumentation with Kubernetes Operator
使用Kubernetes Operator实现自动埋点
yaml
undefinedyaml
undefinedotel-auto-instrumentation.yaml
otel-auto-instrumentation.yaml
Install the OTel Operator first:
Install the OTel Operator first:
helm install opentelemetry-operator open-telemetry/opentelemetry-operator \
helm install opentelemetry-operator open-telemetry/opentelemetry-operator \
--namespace observability --create-namespace
--namespace observability --create-namespace
Define instrumentation for Python services
Define instrumentation for Python services
apiVersion: opentelemetry.io/v1alpha1 kind: Instrumentation metadata: name: python-instrumentation namespace: default spec: exporter: endpoint: http://otel-collector.observability:4317 propagators: - tracecontext - baggage sampler: type: parentbased_traceidratio argument: "0.25" python: image: ghcr.io/open-telemetry/opentelemetry-operator/autoinstrumentation-python:0.44b0 env: - name: OTEL_PYTHON_LOG_CORRELATION value: "true"
apiVersion: opentelemetry.io/v1alpha1 kind: Instrumentation metadata: name: python-instrumentation namespace: default spec: exporter: endpoint: http://otel-collector.observability:4317 propagators: - tracecontext - baggage sampler: type: parentbased_traceidratio argument: "0.25" python: image: ghcr.io/open-telemetry/opentelemetry-operator/autoinstrumentation-python:0.44b0 env: - name: OTEL_PYTHON_LOG_CORRELATION value: "true"
Define instrumentation for Node.js services
Define instrumentation for Node.js services
apiVersion: opentelemetry.io/v1alpha1
kind: Instrumentation
metadata:
name: nodejs-instrumentation
namespace: default
spec:
exporter:
endpoint: http://otel-collector.observability:4317
propagators:
- tracecontext
- baggage
sampler:
type: parentbased_traceidratio
argument: "0.25"
nodejs:
image: ghcr.io/open-telemetry/opentelemetry-operator/autoinstrumentation-nodejs:0.49.1
To instrument a pod, add the annotation:
```yamlapiVersion: opentelemetry.io/v1alpha1
kind: Instrumentation
metadata:
name: nodejs-instrumentation
namespace: default
spec:
exporter:
endpoint: http://otel-collector.observability:4317
propagators:
- tracecontext
- baggage
sampler:
type: parentbased_traceidratio
argument: "0.25"
nodejs:
image: ghcr.io/open-telemetry/opentelemetry-operator/autoinstrumentation-nodejs:0.49.1
要为Pod添加埋点,添加以下注解:
```yamlFor Python:
For Python:
metadata:
annotations:
instrumentation.opentelemetry.io/inject-python: "true"
metadata:
annotations:
instrumentation.opentelemetry.io/inject-python: "true"
For Node.js:
For Node.js:
metadata:
annotations:
instrumentation.opentelemetry.io/inject-nodejs: "true"
undefinedmetadata:
annotations:
instrumentation.opentelemetry.io/inject-nodejs: "true"
undefinedSampling Strategies
采样策略
yaml
undefinedyaml
undefinedTail-based sampling config (in collector)
Tail-based sampling config (in collector)
processors:
tail_sampling:
decision_wait: 10s
num_traces: 100000
policies:
# Always keep error traces
- name: errors
type: status_code
status_code:
status_codes: [ERROR]
# Always keep slow traces (> 2s)
- name: slow-traces
type: latency
latency:
threshold_ms: 2000
# Sample 10% of successful traces
- name: normal-traffic
type: probabilistic
probabilistic:
sampling_percentage: 10
# Always keep traces with specific attributes
- name: important-users
type: string_attribute
string_attribute:
key: user.tier
values: [enterprise, premium]
# Rate limit per service to prevent one service from dominating
- name: rate-limit
type: rate_limiting
rate_limiting:
spans_per_second: 500undefinedprocessors:
tail_sampling:
decision_wait: 10s
num_traces: 100000
policies:
# Always keep error traces
- name: errors
type: status_code
status_code:
status_codes: [ERROR]
# Always keep slow traces (> 2s)
- name: slow-traces
type: latency
latency:
threshold_ms: 2000
# Sample 10% of successful traces
- name: normal-traffic
type: probabilistic
probabilistic:
sampling_percentage: 10
# Always keep traces with specific attributes
- name: important-users
type: string_attribute
string_attribute:
key: user.tier
values: [enterprise, premium]
# Rate limit per service to prevent one service from dominating
- name: rate-limit
type: rate_limiting
rate_limiting:
spans_per_second: 500undefinedBest Practices
最佳实践
- Use tail-based sampling for high-volume production traces.
- Tag telemetry with ,
service.name, andservice.version.deployment.environment - Drop noisy attributes early in the collector.
- Keep metric label cardinality low for stable query performance.
- Use resource detectors to automatically populate cloud metadata.
- Separate collector pools for traces vs metrics if volume requires it.
- Set memory_limiter on every collector pipeline to prevent OOM.
- Use the contrib collector image for production (includes more receivers/exporters).
- 针对高流量生产环境使用基于尾部的采样策略
- 为遥测数据添加、
service.name和service.version标签deployment.environment - 在Collector中尽早过滤冗余属性
- 降低指标标签基数以保证查询性能稳定
- 使用资源探测器自动填充云环境元数据
- 若数据量较大,为链路追踪和指标分别设置独立的Collector池
- 为每个Collector流水线设置memory_limiter以防止内存溢出
- 生产环境使用contrib Collector镜像(包含更多接收器/导出器)
Troubleshooting
故障排查
| Symptom | Check | Fix |
|---|---|---|
| No traces arriving at backend | Collector logs for export errors | Verify endpoint URL and network policy |
| Missing spans in a trace | Propagation headers stripped by proxy | Configure proxy to pass |
| High memory on collector | Too many in-flight traces for tail sampling | Reduce |
| Metric cardinality explosion | Unbounded label values (user IDs, URLs) | Add transform processor to normalize values |
| Auto-instrumentation not working | Pod annotation missing or operator not running | Verify operator is healthy and annotation is correct |
| Duplicate metrics | Both SDK and auto-instrumentation active | Use only one instrumentation method per signal |
| 症状 | 检查项 | 修复方案 |
|---|---|---|
| 后端未收到链路数据 | Collector导出错误日志 | 验证端点URL与网络策略 |
| 链路中缺少Span | 代理剥离了传播头 | 配置代理允许传递 |
| Collector内存占用过高 | 尾部采样的在途链路过多 | 减少 |
| 指标基数爆炸 | 存在无界标签值(如用户ID、URL) | 添加transform处理器标准化值 |
| 自动埋点不生效 | Pod注解缺失或Operator未运行 | 验证Operator健康状态与注解正确性 |
| 指标重复 | SDK与自动埋点同时启用 | 每个信号仅使用一种埋点方式 |
Related Skills
相关技能
- prometheus-grafana - Dashboarding and alerting
- datadog - Managed observability backend
- alerting-oncall - On-call routing and escalation
- rag-observability-evals - RAG-specific observability
- agent-observability - AI agent tracing
- prometheus-grafana - 仪表盘与告警
- datadog - 托管式可观测性后端
- alerting-oncall - 值班路由与升级
- rag-observability-evals - RAG专属可观测性
- agent-observability - AI Agent链路追踪