messaging-streaming

Compare original and translation side by side

🇺🇸

Original

English

🇨🇳

Translation

Chinese

Messaging & Streaming

消息传递与流处理

Move work off the synchronous request path and decouple producers from consumers, so a slow, spiky, or failure-prone operation doesn't block the caller. Getting this wrong is subtle: a queue silently changes the delivery and ordering guarantees, and under load it can absorb a spike gracefully or become the thing that hides a meltdown until the backlog is unrecoverable.

将工作从同步请求路径中移出，解耦生产者与消费者，避免缓慢、峰值波动或易失败的操作阻塞调用方。这部分设计的陷阱很隐蔽：队列会悄然改变投递与顺序保证，在高负载下它既可以平稳吸收流量峰值，也可能成为隐藏系统崩溃的隐患，直到消息积压到无法恢复的地步。

When to reach for this

适用场景

A step is too slow to do inline (image transcode, fan-out, third-party call); the write path is spiky and needs a buffer to smooth bursts (→

back-of-the-envelope

for the spike factor); two services must be decoupled so one can fail or deploy independently; or many consumers need the same event stream. The async hand-off buys responsiveness, isolation, and elasticity (scale producers and consumers separately).

某个步骤耗时过长，无法在线内完成（如图像转码、扇出操作、第三方调用）；写入路径存在流量峰值，需要缓冲区来平滑突发流量（可参考

back-of-the-envelope

计算峰值系数）；两个服务需要解耦，以便其中一个服务可以独立故障或部署；多个消费者需要接收同一事件流。异步移交操作可提升响应性、隔离性与弹性（可分别对生产者和消费者进行扩容）。

When NOT to

不适用场景

The caller needs the result now to proceed (a synchronous read, a balance check before confirming) — a queue only adds latency and a place for work to get lost. Strong read-after-write within one request. Trivial in-process work that a function call handles. Don't add a broker before a number or a coupling problem justifies it (YAGNI): it's a new stateful system to operate, monitor, and reason about under failure. "We'll need Kafka eventually" is name-dropping, not a requirement.

调用方需要立即获取结果才能继续操作（如同步读取、确认前的余额检查）——队列只会增加延迟，还可能导致工作丢失。单次请求内需要强读写一致性。可通过函数调用处理的轻量进程内工作。不要在有明确数据或耦合问题之前就引入消息代理（YAGNI原则）：这会新增一个需要运维、监控并在故障场景下进行分析的有状态系统。“我们最终会用到Kafka”只是随口提及，并非实际需求。

Clarify first

先明确以下问题

Sync or async? Does the caller need the result inline, or is fire-and-react acceptable? This decides whether a queue belongs here at all.
Delivery guarantee needed — is a dropped message acceptable (at-most-once), or must every message be processed (at-least-once + idempotent consumers)?
Ordering — must messages be processed in order, globally or per-key (per user, per account)? Global ordering is expensive; per-key usually suffices.
Throughput and retention — messages/sec at peak, and how long must they be replayable? (→
```
back-of-the-envelope
```
.) One-shot work vs. a replayable log.
Consumer count and pattern — one worker pool draining a job, or many independent subscribers each reading every event?
Failure handling — what happens to a message that keeps failing? Where does it go, and who looks at it?

同步还是异步？ 调用方是否需要在线内获取结果，还是“触发后响应”的模式即可接受？这将决定是否需要引入队列。
所需的投递保证 —— 是否允许消息丢失（at-most-once），还是必须确保每条消息都被处理（at-least-once + 幂等消费者）？
顺序性 —— 消息是否必须按顺序处理，是全局顺序还是按键（如按用户、按账户）排序？全局顺序的成本很高；通常按键排序就足够了。
吞吐量与保留时长 —— 峰值时的消息数/秒，以及消息需要可重放的时长？（可参考
```
back-of-the-envelope
```
。）是一次性工作还是可重放的日志。
消费者数量与模式 —— 是单个工作池处理任务，还是多个独立订阅者各自读取所有事件？
故障处理 —— 持续失败的消息会如何处理？它会被发送到哪里，由谁来检查？

The options

可选方案

Sync vs. async — settle this before picking a tool. Stay synchronous when the caller needs the result to continue and the call is fast and reliable; a direct request is simpler to build, trace, and reason about. Go async when the work is slow, spiky, fan-out-heavy, or the caller can react to the result later — this trades immediate consistency and an easy stack trace for responsiveness and isolation. Only after choosing async do the options below apply. Building request/reply over a queue to fake a synchronous answer is a smell — a direct call is the better design.

Queue (work/task queue) — one logical consumer group competes to drain messages; a message is delivered to one worker and removed when acked. Use when there are background jobs or commands to process exactly once-ish, and workers should scale to drain a backlog.

Pub/sub (fan-out) — each subscriber gets its own copy of every message; producers don't know subscribers. Use when multiple independent consumers react to the same event (notify, index, audit) and loose coupling matters.

Stream (durable, replayable log) — an append-only, partitioned, retained log; consumers track their own offset and can replay history. Use when the design needs ordering per partition, multiple consumers at different positions, event sourcing, or reprocessing (→

data-storage

for event sourcing/outbox).

Durable workflow (orchestration engine) — code that survives process crashes; the engine persists each step and resumes where it left off, with built-in retries, timers, and compensation. Use when a multi-step process with retries, human delays, and rollback (a saga) would otherwise become a fragile hand-rolled mesh of queues, state flags, and cron jobs.

Delivery semantics cut across all of these: at-most-once (fire and forget, may drop), at-least-once (retries until acked, may duplicate — the practical default), exactly-once (no loss, no dup). True end-to-end exactly-once is impractical: a broker's "EOS" (e.g. Kafka) is intra-cluster only, so across systems you always implement it as at-least-once + idempotent/deduped consumers (→

api-design

idempotency keys). See

references/deep-dive.md

for the mechanics.

同步 vs 异步 —— 在选择工具前先确定这一点。 当调用方需要结果才能继续，且调用快速可靠时，保持同步；直接请求的设计更简单，便于追踪和分析。当工作耗时较长、存在流量峰值、扇出操作频繁，或调用方可稍后响应结果时，选择异步——这会以牺牲即时一致性和清晰的堆栈跟踪为代价，换取响应性和隔离性。只有在选择异步后，才考虑以下选项。基于队列构建请求/响应来模拟同步结果是不良设计——直接调用才是更好的方案。

队列（工作/任务队列） —— 一个逻辑消费者组竞争处理消息；消息会被投递到一个工作节点，确认后移除。适用场景：存在需要近乎exactly-once处理的后台任务或命令，且工作节点需要扩容来处理消息积压。

Pub/sub（扇出） —— 每个订阅者都会收到每条消息的副本；生产者不知道订阅者的存在。适用场景：多个独立消费者对同一事件做出反应（通知、索引、审计），且松耦合至关重要。

Stream（持久化、可重放日志） —— 仅追加、分区化、可保留的日志；消费者跟踪自己的偏移量，可重放历史数据。适用场景：设计需要按分区排序、多个消费者处于不同处理位置、事件溯源，或需要重新处理数据（事件溯源/事务性发件箱可参考

data-storage

）。

Durable workflow（编排引擎） —— 可在进程崩溃后继续运行的代码；引擎会持久化每个步骤，并从中断处恢复，内置重试、定时器和补偿机制。适用场景：多步骤流程包含重试、人工延迟和回滚（Saga），否则会变成由队列、状态标志和定时任务组成的脆弱手动架构。

投递语义适用于所有上述方案：at-most-once（即发即弃，可能丢失消息）、at-least-once（重试直到确认，可能重复——实际默认方案）、exactly-once（无丢失、无重复）。真正的端到端exactly-once并不现实：消息代理的“EOS”（如Kafka）仅适用于集群内，因此跨系统实现时，始终采用at-least-once + 幂等/去重消费者的方案（可参考

api-design

中的幂等键）。有关机制细节，请查看

references/deep-dive.md

。

Trade-offs

权衡取舍

Option	What it solves	What it worsens	Change it when
Queue (work queue)	Decouples + buffers; scale workers to drain backlog	At-least-once means duplicates; ordering not guaranteed across workers	Replay or many independent consumers are needed → stream/pub-sub
Pub/sub (fan-out)	One event, N decoupled reactions; add consumers freely	No replay (transient); slow subscriber can lag or drop; fan-out amplifies load	History/replay or per-key ordering is needed → stream
Stream (log)	Ordering per partition, replay, multi-consumer, event sourcing	Operationally heavier; partition key is a hot-shard risk; consumers must manage offsets	Simple one-shot jobs don't need a log → queue
Durable workflow	Crash-safe long-running sagas; retries/compensation built in	New runtime + programming model; latency overhead; lock-in to engine semantics	A single async step with no orchestration → plain queue
At-least-once delivery	No message loss under retry/crash	Duplicates — consumers must be idempotent (→ `api-design` idempotency keys)	Loss is acceptable and dedup cost isn't worth it → at-most-once
Exactly-once (effective)	No loss and no duplicate side effects	Cost/complexity; often narrow (within one broker, not across systems)	Idempotent at-least-once is good enough (it usually is)

选项	解决的问题	带来的问题	何时更换
队列（工作队列）	解耦 + 缓冲；扩容工作节点处理积压	at-least-once会导致重复；跨工作节点无法保证顺序	需要重放或多个独立消费者 → stream/pub-sub
Pub/sub（扇出）	一个事件触发N个解耦的反应；可自由添加消费者	无法重放（临时消息）；慢速订阅者可能滞后或丢失消息；扇出会放大负载	需要历史/重放或按键排序 → stream
Stream（日志）	按分区排序、可重放、多消费者、事件溯源	运维复杂度更高；分区键存在热点分片风险；消费者必须管理偏移量	简单的一次性任务不需要日志 → 队列
Durable workflow	崩溃安全的长期运行Saga；内置重试/补偿机制	新增运行时 + 编程模型；延迟开销；依赖引擎语义	无需编排的单个异步步骤 → 普通队列
at-least-once投递	重试/崩溃场景下无消息丢失	重复消息——消费者必须实现幂等（参考 `api-design` 的幂等键）	允许消息丢失且去重成本过高 → at-most-once
exactly-once（有效）	无丢失且无重复副作用	成本/复杂度高；通常适用范围窄（仅单个代理内，跨系统不适用）	幂等的at-least-once已足够（通常如此）

Behavior under stress

压力下的表现

A broker's whole job is to absorb a spike — but it can also hide a meltdown.

Backlog growth / unbounded queues: producers outpace consumers; the queue grows past memory, spilling to disk and slowing further. End-to-end latency climbs invisibly while throughput looks fine. Mitigate: backpressure — bound the queue and shed or 503 producers (with backoff) once full, rather than buffering forever. Alarm on queue depth and message age, not just rate.
Retry storms / poison messages: a message that always fails is redelivered forever (at-least-once), burning consumer capacity and re-amplifying load on a downstream that's already struggling. Mitigate: capped retries with backoff+jitter, then route to a dead-letter queue (DLQ) so the poison message stops blocking the line and a human can inspect it. (Retries, backoff, DLQ-as-containment, and backpressure are owned by
```
resilience-failure
```
.)
Duplicate amplification: under retry, the same side effect (charge, email) fires twice unless consumers dedup. Mitigate: idempotency keys / dedup table.
Hot partition: in a stream, a skewed partition key (one celebrity, one tenant) overloads a single partition while others idle — the same hot-key shape as
```
data-storage
```
sharding. Mitigate: better key, sub-partitioning, or batching.
Slow consumer in fan-out: one lagging subscriber backs up or drops; isolate consumers so one can't stall the others.

Monitor: queue depth, oldest-message age / consumer lag (the single best signal), redelivery/DLQ rate, consumer throughput vs. producer rate, and end-to-end latency.

消息代理的核心作用是吸收流量峰值，但它也可能隐藏系统崩溃。

消息积压增长 / 无界队列：生产者速度超过消费者；队列内存耗尽后写入磁盘，进一步变慢。端到端延迟无形上升，而吞吐量看似正常。缓解措施：背压——限制队列大小，队列满时拒绝或返回503给生产者（带退避机制），而非无限缓冲。监控队列深度和消息时长，而非仅监控速率。
重试风暴 / 有毒消息：持续失败的消息会被无限重传（at-least-once），消耗消费者资源，并给已陷入困境的下游系统带来更大负载。缓解措施：设置重试次数上限并采用退避+抖动策略，然后将有毒消息路由到死信队列（DLQ），避免其阻塞正常消息流，同时便于人工检查。（重试、退避、DLQ隔离和背压属于
```
resilience-failure
```
的范畴。）
重复消息放大：重试时，相同的副作用（如收费、发送邮件）会触发两次，除非消费者实现去重。缓解措施：使用幂等键 / 去重表。
热点分区：在stream中，倾斜的分区键（如某个知名用户、某个租户）会导致单个分区过载，而其他分区闲置——这与
```
data-storage
```
分片的热点键问题相同。缓解措施：优化键设计、子分区或批处理。
扇出中的慢速消费者：一个滞后的订阅者会导致消息积压或丢失；隔离消费者，避免单个消费者阻塞其他消费者。

监控指标：队列深度、最旧消息时长 / 消费者滞后（最佳信号）、重传/DLQ率、消费者吞吐量 vs 生产者速率、端到端延迟。

How to apply

实施步骤

Clarify the inputs. Settle sync vs. async first; if the caller needs the result inline, stop — no broker. Then answer delivery guarantee, ordering scope, throughput/retention, consumer pattern, and failure handling (see Clarify first).
Pick the shape from the trade-off table. One worker pool draining jobs → queue; many independent reactions to one event → pub/sub; ordering, replay, or multiple offsets → stream; multi-step retry/compensation saga → durable workflow. Pick the cheapest shape that meets the constraint.
Set the key knobs. Choose the delivery semantic (at-least-once + idempotent consumers is the default), the ack point (after-process vs. before), the partition/ordering key, retention window, and retry cap before the DLQ.
Stress-test the choice. Walk backlog growth, retry storms / poison messages, duplicate amplification, hot partition, and slow fan-out consumer. Add backpressure, a DLQ, and dedup where each applies.
Size it with numbers. Compute peak produce vs. sustained consume rate,
```
consume_rate − produce_rate
```
drain, partition count vs. consumer parallelism, and storage = rate × message size × retention (→
```
back-of-the-envelope
```
).
Pick a provider. Default to the generic recipe; only read a provider file if the user names a cloud and a managed limit changes the choice.

明确输入条件：先确定同步还是异步；如果调用方需要在线内获取结果，直接停止——无需消息代理。然后确定投递保证、顺序范围、吞吐量/保留时长、消费者模式和故障处理（参考「先明确以下问题」部分）。
根据权衡表选择类型：单个工作池处理任务 → 队列；一个事件触发多个独立反应 → pub/sub；需要排序、重放或多个偏移量 → stream；多步骤重试/补偿Saga → Durable workflow。选择满足约束条件的最简方案。
设置关键参数：选择投递语义（默认at-least-once + 幂等消费者）、确认时机（处理后 vs 处理前）、分区/排序键、保留窗口、进入DLQ前的重试上限。
压力测试所选方案：模拟消息积压增长、重试风暴/有毒消息、重复消息放大、热点分区、扇出慢速消费者等场景。在适用的场景添加背压、DLQ和去重机制。
量化规模：计算峰值生产速率 vs 持续消费速率、
```
消费速率 − 生产速率
```
的积压处理速度、分区数 vs 消费者并行度、存储量 = 速率 × 消息大小 × 保留时长（参考
```
back-of-the-envelope
```
）。
选择提供商：默认采用通用方案；仅当用户指定云平台且托管服务的限制会改变选择时，才查看提供商文件。如果该提供商没有对应的文件，则通用方案即为答案。

Dos and don'ts

注意事项

Settle sync vs. async before naming any broker; keep synchronous work synchronous.
Default to at-least-once and make consumers idempotent (→
```
api-design
```
keys).
Bound queues and apply backpressure; alarm on oldest-message age / consumer lag.
Cap retries with backoff+jitter, then route poison messages to a DLQ.
Define the message envelope (
```
message_id
```
, schema version, key,
```
trace_id
```
) up front.

Don't

Don't add a broker "for scale later" before a number or coupling problem justifies it.
Don't build request/reply over a queue to fake a synchronous answer.
Don't reach for exactly-once when idempotent at-least-once already suffices.
Don't buffer an unbounded backlog — it hides a meltdown until it's unrecoverable.
Don't pick a global ordering guarantee when per-key ordering is enough.

应当

在提及任何消息代理前先确定同步还是异步；保持同步工作的同步性。
默认采用at-least-once并让消费者实现幂等（参考
```
api-design
```
的键）。
限制队列大小并应用背压；监控最旧消息时长/消费者滞后并设置告警。
设置重试次数上限并采用退避+抖动策略，然后将有毒消息路由到DLQ。
提前定义消息信封（
```
message_id
```
、模式版本、键、
```
trace_id
```
）。

不应当

在有明确数据或耦合问题之前，不要为了“日后扩容”而引入消息代理。
不要基于队列构建请求/响应来模拟同步结果。
当幂等的at-least-once已足够时，不要追求exactly-once。
不要缓冲无界的消息积压——这会隐藏系统崩溃，直到无法恢复。
当按键排序足够时，不要选择全局顺序保证。

Numbers that matter

关键量化指标

Quantify before choosing: peak produce rate vs. sustained consume rate (if producers can outrun consumers for long, backpressure and a depth alarm are required), retention window (drives storage = rate × message size × retention), and partition count (caps consumer parallelism — one consumer per partition per group). A backlog drains at

(consume_rate − produce_rate)

; if that's negative, it never drains. Use

back-of-the-envelope

for the spike factor, message sizes, and storage; don't restate its tables here.

选择前先量化：峰值生产速率 vs 持续消费速率（如果生产者长期快于消费者，则需要背压和深度告警）、保留窗口（决定存储量 = 速率 × 消息大小 × 保留时长）、分区数（限制消费者并行度——每个分组每个分区对应一个消费者）。消息积压的处理速度为

(消费速率 − 生产速率)

；如果该值为负，则积压永远无法处理。使用

back-of-the-envelope

计算峰值系数、消息大小和存储量；无需在此重复其表格内容。

Interface sketch

接口示例

A message is a contract. Define it explicitly, not as "an event":

Envelope: stable
```
message_id
```
(for dedup),
```
type
```
/
```
schema_version
```
, a
```
partition_key
```
/ordering key,
```
timestamp
```
, and a
```
trace_id
```
for correlation.
Payload: a versioned schema. Prefer event facts (
```
OrderPlaced{order_id, total}
```
) over commands when fanning out; keep it small and forward-compatible.
Ack contract: when does the consumer ack — before or after the side effect? Ack-after-process gives at-least-once; ack-before gives at-most-once.
DLQ shape: failed messages keep the original envelope plus failure reason and attempt count, so they can be inspected and replayed.

消息是一种契约。需明确定义，而非简单称为“事件”：

信封：稳定的
```
message_id
```
（用于去重）、
```
type
```
/
```
schema_version
```
、
```
partition_key
```
/排序键、
```
timestamp
```
、用于关联的
```
trace_id
```
。
负载：带版本的模式。扇出时优先使用事件事实（如
```
OrderPlaced{order_id, total}
```
）而非命令；保持负载小巧且向前兼容。
确认契约：消费者何时确认——处理前还是处理后？处理后确认对应at-least-once；处理前确认对应at-most-once。
DLQ格式：失败消息保留原始信封，加上失败原因和尝试次数，以便检查和重放。

Choosing a provider

选择提供商

Default to the generic recipe above. If the user names a cloud, read

references/providers/<provider>.md

for the managed-service mapping, quotas/limits, and provider-specific trade-offs. If no file exists for that provider, the generic recipe is the answer.

默认采用上述通用方案。如果用户指定云平台，请查看

references/providers/<provider>.md

获取托管服务映射、配额/限制和提供商特定的权衡。如果该提供商没有对应的文件，则通用方案即为答案。

Diagram

图表

To visualize the producer → broker → consumer path, fan-out to multiple subscribers, or the retry → DLQ flow, use the in-plugin

architecture-diagram

skill. Quick inline sketch:

producer → [queue] → workers ─fail→ [DLQ]

; the main path solid, the DLQ branch dashed.

如需可视化生产者→代理→消费者路径、扇出到多个订阅者或重试→DLQ流程，请使用插件内的

architecture-diagram

Skill。快速内联示意图：

producer → [queue] → workers ─fail→ [DLQ]

；主路径为实线，DLQ分支为虚线。

Related building blocks

References

参考资料

references/deep-dive.md
— delivery-guarantee mechanics (acks, visibility timeouts, offsets, idempotent/dedup consumers, transactional outbox), ordering internals, partitioning, DLQ design, and when durable-workflow engines beat hand-rolled queue+retry+saga. Read when designing the messaging layer in detail.
references/providers/{generic,aws,azure,gcp,temporal}.md
— service mappings, decision-changing limits, and pitfalls per environment.

references/deep-dive.md
—— 投递保证机制（确认、可见性超时、偏移量、幂等/去重消费者、事务性发件箱）、顺序内部原理、分区、DLQ设计，以及何时Durable workflow引擎优于手动构建的队列+重试+Saga。在详细设计消息层时阅读。
references/providers/{generic,aws,azure,gcp,temporal}.md
—— 服务映射、会改变决策的限制、各环境的陷阱。