distributed-logging

Compare original and translation side by side

🇺🇸

Original

English

🇨🇳

Translation

Chinese

Distributed logging

分布式日志

Move logs from thousands of processes into one searchable place, fast enough to debug a live incident and cheap enough to keep for months. Getting it wrong is a classic "ignore failure" miss: the logging pipeline is itself a distributed system that buckles under the exact traffic spike you most need it during, and a naive design either drops the evidence or takes down the app it instruments.

将数千个进程产生的日志集中到一个可搜索的位置，既要快到能调试实时事件，又要足够经济以留存数月。设计失误是典型的「忽视故障」问题：日志流水线本身就是一个分布式系统，会在你最需要它的流量峰值时崩溃，而朴素设计要么丢失关键证据，要么拖垮它所监控的应用。

When to reach for this

适用场景

More than one process emits logs and someone needs to search them together; an incident requires correlating a request across services; log volume has outgrown

grep

on a box; or compliance demands retention. The pipeline buys central search, cross-service correlation, and a durable record decoupled from any single host.

多个进程产生日志且需要统一搜索；事件排查需要跨服务关联请求；日志量已超出单主机

grep

的处理能力；合规要求日志留存。该流水线提供集中搜索、跨服务关联能力，以及与单一主机解耦的持久化记录。

When NOT to

不适用场景

A single service on one host where

journald

+ log rotation is enough — a full pipeline is pure operational overhead (YAGNI). Numeric time-series questions ("what is p99 latency", "is error rate up") belong to metrics, not log scans — that is

observability

's job; logs answer "what exactly happened to this request". Don't ship every debug line at full volume before a number shows the volume justifies the cost; sample first.

单主机上的单一服务，使用

journald

+日志轮转即可满足需求——完整流水线纯粹是运维开销（YAGNI）。数值型时间序列问题（如「p99延迟是多少」「错误率是否上升」）属于metrics范畴，而非日志扫描——这是

observability

的职责；日志用于解答「这个请求到底发生了什么」。在确认日志量值得投入成本前，不要全量传输所有调试日志；应先进行采样。

Clarify first

前期澄清要点

Volume and peak — lines/sec and bytes/sec, average and peak (→
```
back-of-the-envelope
```
). This sizes every stage.
Structured or free-text — can producers emit JSON now, or is there legacy text to parse?
Query latency need — interactive search in seconds (hot index) vs. occasional forensic/audit reads (cold archive)?
Retention + compliance — how long hot, how long cold, any legal hold or PII redaction requirement?
Loss tolerance — may logs be dropped/sampled under overload, or is every line evidence (audit/financial)?

日志量与峰值——每秒行数、每秒字节数，平均值与峰值（参考
```
back-of-the-envelope
```
）。这决定了每个阶段的规模。
结构化还是自由文本——日志生产者当前能否输出JSON，还是存在需要解析的遗留文本？
查询延迟需求——是需要秒级交互式搜索（热索引），还是偶尔的取证/审计读取（冷归档）？
留存与合规——热数据留存多久，冷数据留存多久，是否有法定保留或PII脱敏要求？
丢失容忍度——过载时是否允许丢弃/采样日志，还是每一行日志都是审计/财务证据？

The options

可选方案

Collection (agent on the host)

Sidecar/node agent (Fluentd, Fluent Bit, Vector, Filebeat): tails files or reads stdout, adds metadata, ships out. Use when apps log to files/stdout and you want app code untouched — the default.
Direct-to-bus SDK: the app writes structured events straight to a transport. Use when you control the code and want exact structure, accepting tighter coupling.

Transport / buffer (the shock absorber)

Agent-side buffer + direct ship to indexer: simplest; agent disk-buffers and retries. Use at low-to-moderate volume with one consumer.
Durable log bus (
messaging-streaming
, e.g. Kafka): producers write to a partitioned bus; indexers consume at their own pace. Use at high volume or when multiple sinks (search, archive, analytics) read the same stream.

Index / store (the search backend)

Full-text index (Elasticsearch/OpenSearch — the "E" in ELK/EFK): rich queries, expensive RAM/disk. Use when interactive field search matters.
Label-indexed store (Loki): indexes only labels, stores log bodies compressed; much cheaper, grep-style queries. Use for high volume where you mostly filter by service/label then scan.

Retention / tiering

Hot index → cold object store (
blob-store
): keep days of searchable data hot, roll older data to compressed objects. Use whenever retention exceeds the hot window economically (almost always).

收集（主机上的agent）

Sidecar/节点agent（Fluentd、Fluent Bit、Vector、Filebeat）： 监听文件或读取stdout，添加元数据后传输。适用于应用日志写入文件/stdout且希望不修改应用代码的场景——这是默认方案。
直接写入总线SDK： 应用将结构化事件直接写入传输层。适用于你控制代码且需要精确结构的场景，但会增加耦合度。

传输/缓冲（减震器）

Agent端缓冲+直接传输至索引器： 最简单的方案；Agent通过磁盘缓冲并重试。适用于低至中等日志量且只有一个消费者的场景。
持久化日志总线（
messaging-streaming
，如Kafka）：生产者写入分区总线；索引器按自身速度消费。适用于高日志量或多个 sink（搜索、归档、分析）读取同一流的场景。

索引/存储（搜索后端）

全文索引（Elasticsearch/OpenSearch——ELK/EFK中的「E」）： 支持丰富查询，但RAM/磁盘成本高。适用于交互式字段搜索很重要的场景。
标签索引存储（Loki）： 仅索引标签，日志体压缩存储；成本低得多，支持类grep查询。适用于高日志量且主要通过服务/标签过滤后扫描的场景。

留存/分层

热索引→冷对象存储（
blob-store
）：将可搜索数据保留数天作为热数据，旧数据滚动至压缩对象存储。几乎所有场景下，当留存时间超出热窗口的经济阈值时都适用。

Trade-offs

权衡分析

Option	What it solves	What it worsens	Change it when
Node agent (Fluent Bit/Vector)	No app changes; central metadata + routing	One more daemon per host; parsing CPU; agent can lag	You need exact structure → emit structured events from the app
Direct-to-bus SDK	Clean structured events, no file parse	Couples app to transport; app blocks/loses logs if bus is down	Coupling/availability hurts → go back to agent + local buffer
Agent buffer → direct indexer	Fewest moving parts	Indexer backpressure hits producers; no replay; one sink	Volume spikes or you need >1 sink → add a log bus
Durable log bus (Kafka)	Absorbs spikes, decouples, replay, fan-out	Extra system to run; ordering only per-partition; cost	Volume is low and single-sink → drop the bus
Full-text index (ES/OpenSearch)	Fast rich field search	RAM/disk hungry; mapping explosions; costly at scale	Cost dominates and queries are label-filtered → Loki
Label-indexed (Loki)	Cheap storage at high volume	Weak full-text; slow on high-cardinality scans	You truly need arbitrary field search → full-text index
Hot index + cold archive	Cheap long retention	Cold reads are slow/manual to rehydrate	Forensic reads on old data must be fast → widen hot window

选项	解决的问题	带来的问题	何时更换
节点agent（Fluent Bit/Vector）	无需修改应用；集中元数据与路由	每台主机多一个守护进程；解析占用CPU；agent可能滞后	需要精确结构→从应用输出结构化事件
直接写入总线SDK	结构化事件清晰，无需文件解析	应用与传输层耦合；总线故障时应用可能阻塞/丢失日志	耦合/可用性问题影响严重→退回agent+本地缓冲
Agent缓冲→直接索引器	最少的移动部件	索引器背压会影响生产者；无法重放；仅支持一个sink	出现流量峰值或需要多个sink→添加日志总线
持久化日志总线（Kafka）	吸收流量峰值，解耦组件，支持重放与扇出	额外的运维系统；仅分区内有序；成本较高	日志量低且仅需单一sink→移除总线
全文索引（ES/OpenSearch）	快速的丰富字段搜索	消耗大量RAM/磁盘；映射爆炸；大规模场景成本高	成本占主导且查询以标签过滤为主→改用Loki
标签索引（Loki）	高日志量下存储成本低	全文搜索能力弱；高基数扫描速度慢	确实需要任意字段搜索→改用全文索引
热索引+冷归档	低成本长期留存	冷数据读取慢/需手动恢复	旧数据的取证读取需要快速响应→扩大热窗口

Behavior under stress

压力下的表现

The pipeline's failure mode is that incidents generate log spikes — an outage emits floods of errors and stack traces exactly when the pipeline is busiest, so it must degrade without amplifying the outage.

Producer backpressure: if the indexer slows and agents ship synchronously, log calls can block the app. Mitigate: bounded local buffer with drop-newest / sample on overflow, never block the request path. A durable bus moves the backlog off the hosts (backpressure + DLQ semantics are owned by
```
messaging-streaming
```
).
Volume amplification: one bad deploy logging at debug can 100× volume and blow the index. Mitigate: per-service rate caps at the agent (rate limiting is owned by
```
resilience-failure
```
), dynamic sampling, alerts on ingest bytes/sec.
Index hot-shard / mapping explosion: high-cardinality fields (raw user IDs as index fields) or one fat shard wrecks the cluster. Mitigate: time-based indices, bounded field mappings, label discipline.
Ordering: the bus only orders within a partition; merged multi-host streams interleave. Mitigate: sort by event timestamp at query time; carry a monotonic sequence or trace span order, don't trust arrival order.
Losing the evidence: dropping logs silently hides the very failure under investigation. Mitigate: meter and alert on drop/sample rate so loss is visible.

Monitor: ingest bytes/sec and lines/sec, end-to-end ship lag, bus consumer lag, agent buffer fullness + drop rate, index queue/rejection rate, query latency.

流水线的故障模式在于事件会触发日志峰值——故障会产生大量错误和堆栈跟踪，而此时正是流水线最繁忙的时候，因此它必须在不加剧故障的前提下优雅降级。

生产者背压： 如果索引器变慢且agent同步传输，日志调用可能阻塞应用。缓解方案： 采用有界本地缓冲，溢出时丢弃最新日志/采样，绝不要阻塞请求路径。持久化总线将积压转移至主机外（背压+DLQ语义由
```
messaging-streaming
```
负责）。
日志量激增： 一次错误的调试部署可能使日志量增加100倍，撑爆索引。缓解方案： 在agent端设置每个服务的速率上限（速率限制由
```
resilience-failure
```
负责），动态采样，对摄入字节/秒设置告警。
索引热分片/映射爆炸： 高基数字段（如将原始用户ID作为索引字段）或过大的分片会破坏集群。缓解方案： 基于时间的索引，有界字段映射，规范标签使用。
顺序问题： 总线仅保证分区内有序；多主机流合并后会交错。缓解方案： 查询时按事件时间戳排序；携带单调序列或trace span顺序，不要依赖到达顺序。
证据丢失： 静默丢弃日志会隐藏正在排查的故障。缓解方案： 计量并告警丢弃/采样率，使丢失情况可见。

监控指标： 摄入字节/秒与行数/秒，端到端传输延迟，总线消费者延迟，agent缓冲填充率+丢弃率，索引队列/拒绝率，查询延迟。

How to apply

实施步骤

Clarify the inputs — pin volume (avg + peak bytes/sec), structured-vs-text, query-latency need, retention, and loss tolerance (see Clarify first). If volume is tiny and single-host, stop — host-local logs suffice (YAGNI).
Pick stages from the trade-off table — choose a collector, decide whether a durable bus is warranted (volume/spike/multi-sink), pick the index backend by the query need, and choose the hot window + cold tier.
Set the key knobs — mandate a structured schema with a correlation/trace ID on every line, set agent buffer size + overflow policy, sampling rates, index rollover (time-based) and shard sizing, and the retention/tiering policy.
Stress-test the choice — walk each item in Behavior under stress (producer backpressure, a debug-flood deploy, hot shard, ordering, silent drops) and confirm the pipeline degrades by sampling/buffering, never by blocking the app.
Size it with numbers — multiply lines/sec × bytes/line × peak factor × retention to get hot index GB and cold archive GB/month; confirm the bus partition count and indexer fleet cover peak ingest (→ Numbers that matter).
Pick a provider — default to the generic recipe; open a provider file only if the user named a cloud (see Choosing a provider).

明确输入条件——确认日志量（平均+峰值字节/秒）、结构化/文本类型、查询延迟需求、留存时间和丢失容忍度（见「前期澄清要点」）。如果日志量极小且为单主机场景，停止实施——主机本地日志已足够（YAGNI）。
根据权衡表选择组件——选择收集器，判断是否需要持久化总线（日志量/峰值/多sink），根据查询需求选择索引后端，确定热窗口+冷存储层。
设置关键参数——强制要求每条日志包含带correlation/trace ID的结构化 schema，设置agent缓冲大小+溢出策略、采样率、索引滚动（基于时间）和分片大小，以及留存/分层策略。
压力测试选型——逐一验证「压力下的表现」中的各项（生产者背压、调试日志激增、热分片、顺序问题、静默丢弃），确认流水线通过采样/缓冲降级，绝不阻塞应用。
量化规模——用行数/秒 × 单日志字节数 × 峰值系数 × 留存时间计算热索引GB数和每月冷归档GB数；确认总线分区数和索引器集群能覆盖峰值摄入（参考
```
back-of-the-envelope
```
）。
选择服务商——默认使用通用方案；仅当用户指定云服务商时，查看
```
references/providers/<provider>.md
```
获取托管服务映射、配额/限制和服务商特定权衡。如果该服务商无对应文件，通用方案即为答案。

Dos and don'ts

注意事项

Emit structured logs (JSON) with a correlation/trace ID stamped at the edge and propagated, so a request is one query.
Put a bounded buffer between producers and the indexer and drop/sample on overflow rather than block the app.
Use a durable log bus once volume spikes or more than one sink reads the stream.
Tier aggressively: short hot window for search, compressed cold archive for retention.
Meter ingest rate, ship lag, and drop rate, and alert on them — invisible loss is the trap.

Don't

Don't let a logging call block or crash the request path; logging is best-effort by default.
Don't index high-cardinality free-form fields — it explodes the cluster; keep labels bounded.
Don't trust arrival order for causality; sort by event time / sequence (→
```
sequencer
```
).
Don't re-teach metrics/alerting here or set SLOs in logs — that's
```
observability
```
.
Don't keep everything hot forever to avoid cold reads; size the hot window to the debugging window.

建议

输出结构化日志（JSON），在边缘打上correlation/trace ID并传播，使单个请求可通过一次查询关联。
在生产者和索引器之间设置有界缓冲，溢出时丢弃/采样而非阻塞应用。
当日志量出现峰值或多个sink读取流时，使用持久化日志总线。
积极分层：短热窗口用于搜索，压缩冷归档用于留存。
计量摄入速率、传输延迟和丢弃率并设置告警——不可见的丢失是陷阱。

禁忌

不要让日志调用阻塞或崩溃请求路径；默认日志是尽力而为的操作。
不要索引高基数自由格式字段——这会撑爆集群；保持标签有界。
不要依赖到达顺序判断因果关系；按事件时间/序列排序（参考
```
sequencer
```
）。
不要在此重复metrics/告警内容或在日志中设置SLO——这是
```
observability
```
的职责。
不要为避免冷读取而永久保留所有热数据；根据调试窗口设置热窗口大小。

Numbers that matter

关键数值

Size the pipeline from volume: lines/sec × bytes/line gives ingest bytes/sec; a few KB/line at tens of thousands of lines/sec is already 100s of MB/s and TBs/day. Hot index storage ≈ daily bytes × hot-days × (1 + replica + overhead); cold archive ≈ daily bytes × retention-days, compressed ~5–10×. Apply a peak multiplier for incident floods. Don't restate latency/QPS/storage rules of thumb here — pull them from

back-of-the-envelope

根据日志量确定流水线规模：行数/秒 × 单日志字节数得出摄入字节/秒；每条日志几KB、每秒数万行的规模已达数百MB/s、每日TB级。热索引存储≈每日字节数 × 热数据天数 × (1 + 副本 + 开销)；冷归档≈每日字节数 × 留存天数，压缩比约5–10倍。为事件激增应用峰值系数。此处不重复延迟/QPS/存储经验法则——参考

back-of-the-envelope

。

Interface sketch

接口示例

A log line is a contract: a structured event, not a string. Minimum fields:

{ "ts": "2026-05-29T12:00:00.123Z", "level": "ERROR", "service": "checkout",
  "host": "pod-7", "trace_id": "abc123", "span_id": "f9", "msg": "charge failed",
  "user_id": "u42", "err": "timeout", "latency_ms": 812 }

ts

is the event time (sort key at query);

trace_id

span_id

correlate across services;

level

and

service

are bounded index labels; high-cardinality values (

user_id

) stay as searchable body fields, not index keys. The bus partition key is usually

service

trace_id

to keep a request's lines ordered together.

日志行是一种契约：是结构化事件，而非字符串。最小字段集：

{ "ts": "2026-05-29T12:00:00.123Z", "level": "ERROR", "service": "checkout",
  "host": "pod-7", "trace_id": "abc123", "span_id": "f9", "msg": "charge failed",
  "user_id": "u42", "err": "timeout", "latency_ms": 812 }

ts

是事件时间（查询时的排序键）；

trace_id

span_id

用于跨服务关联；

level

和

service

是有界索引标签；高基数值（如

user_id

）作为可搜索的体字段，而非索引键。总线分区键通常为

service

或

trace_id

，以保持同一请求的日志行有序。

Choosing a provider

选择服务商

Default to the generic recipe above. If the user names a cloud, read

references/providers/<provider>.md

for the managed-service mapping, quotas/limits, and provider-specific trade-offs. If no file exists for that provider, the generic recipe is the answer.

默认使用上述通用方案。如果用户指定云服务商，查看

references/providers/<provider>.md

获取托管服务映射、配额/限制和服务商特定权衡。如果该服务商无对应文件，通用方案即为答案。

Diagram

图表

To visualize the pipeline (producers → agents → bus → indexer/search + archive sink) or the overflow/degradation path, use the in-plugin

architecture-diagram

skill — draw the durable bus as the buffer between producers and sinks, the cold-archive sink with a

blob-store

color, and the drop-on-overflow path as a dashed arrow.

如需可视化流水线（生产者→agents→总线→索引器/搜索+归档sink）或溢出/降级路径，使用插件内的

architecture-diagram

技能——将持久化总线绘制为生产者与sink之间的缓冲，冷归档sink使用

blob-store

颜色，溢出丢弃路径用虚线箭头表示。

Related building blocks

References

参考资料

references/deep-dive.md
— buffering and backpressure mechanics, sampling strategies, structured-logging + correlation-ID propagation, index lifecycle/rollover, ordering, and cold tiering. Read when designing the pipeline in detail.
references/providers/{generic,aws,azure,gcp}.md
— service mappings, limits, and pitfalls per environment.

references/deep-dive.md
——缓冲与背压机制、采样策略、结构化日志+correlation ID传播、索引生命周期/滚动、顺序处理和冷分层。详细设计流水线时阅读。
references/providers/{generic,aws,azure,gcp}.md
——各环境下的服务映射、限制和注意事项。