sharded-counters

Compare original and translation side by side

🇺🇸

Original

English

🇨🇳

Translation

Chinese

Sharded counters

分片计数器（Sharded counters）

Count a thing that is incremented far faster than a single row, key, or partition can serialize writes — likes, views, votes, rate tallies, inventory decrements. The trap is the hot counter: every writer contends on one record, so latency climbs and throughput plateaus no matter how big the box is. Getting it wrong turns a trivial

+1

into the bottleneck of the whole feature.

当某一数据的增量速度远超单行、单键或单分区的写入序列化能力时（例如点赞数、浏览量、投票数、速率统计、库存递减），可采用本方案。这里的陷阱是热点计数器：所有写操作都争抢同一条记录，无论服务器配置多高，延迟都会上升，吞吐量也会停滞不前。如果处理不当，一个简单的

+1

操作会成为整个功能的瓶颈。

When to reach for this

适用场景

Concurrent increments to a single logical count exceed what one row/key can absorb — a viral post's like count, a live-event view counter, a global rate tally. The symptom is write contention (lock waits, CAS retries, partition hot-spotting) on one record while the rest of the store is idle. Reaching for this means the write side is the problem, and an exact-to-the-millisecond total is not required.

对单个逻辑计数的并发增量超出了单行/单键的处理能力——比如爆款帖子的点赞数、直播活动的浏览计数器、全局速率统计。典型症状是：某一条记录出现写竞争（锁等待、CAS重试、分区热点），而存储系统的其他部分却处于空闲状态。采用本方案意味着写侧是核心问题，且不需要精确到毫秒级的实时总数。

When NOT to

不适用场景

Low write rate (a single atomic

INCR

handles thousands/sec — don't shard a counter nobody is hammering; YAGNI). Counts that must be transactionally exact and read-after-write consistent at every instant (bank balances, seat inventory at sell-out) — that's a transactional decrement, see

consistency-coordination

, not a fan-out tally. Counting distinct items exactly (unique visitors) where you also need the member list — that's a set in the store, not a counter. If reads dominate and writes are cheap, you need a cached aggregate, not sharding.

低写入速率：单个原子
```
INCR
```
操作每秒可处理数千次请求——不要对无人频繁操作的计数器进行分片，遵循YAGNI（You Aren't Gonna Need It，不要过度设计）原则。
需要事务级精确性且随时保持读写一致性的计数（如银行余额、售罄状态下的库存）——这属于事务性递减操作，请参考
```
consistency-coordination
```
，而非扇出统计方案。
需要精确统计唯一项目（如独立访客）且同时需要成员列表的场景——这应使用存储系统中的集合，而非计数器。
读操作占主导、写操作成本低廉的场景：此时需要的是缓存聚合结果，而非分片。

Clarify first

先明确以下问题

Write rate to the hottest single count — peak increments/sec on one logical counter, not the aggregate (→
```
back-of-the-envelope
```
).
Exact or approximate — is an off-by-a-few total acceptable, and for how long may shards disagree (eventual)? Drives shard count and read path.
Counting occurrences or distinct items — a running total vs. unique-count (likes vs. unique viewers) decides plain shards vs. HyperLogLog.
Read rate and freshness — how often is the total read, and how stale may the served number be (sub-second? minutes?).
Time-windowed or lifetime — "views in the last hour" needs bucketed keys and expiry; a lifetime total does not.

最热单个计数的写入速率：单个逻辑计数的峰值增量/秒，而非总增量（参考
```
back-of-the-envelope
```
）。
精确还是近似：总数允许少量误差吗？分片之间的不一致可容忍多长时间（最终一致性）？这将决定分片数量和读取路径。
统计出现次数还是唯一项目：累计总数 vs 唯一计数（点赞数 vs 独立浏览者）决定了使用普通分片还是HyperLogLog。
读取速率和新鲜度：总数的读取频率如何？允许返回的数据有多陈旧（亚秒级？分钟级？）。
时间窗口还是生命周期：“过去一小时的浏览量”需要按时间分桶的键和过期机制；生命周期总数则不需要。

The options

可选方案

Single atomic counter — one row/key with atomic
```
INCR
```
/
```
UPDATE +1
```
. Use when peak write rate on the hottest count is well within one node's serialized write throughput. The default; don't outgrow it prematurely.
Write-sharded (striped) counter — split one logical count into N physical shards (
```
counter:{id}:shard:{0..N-1}
```
); each write increments a random/hashed shard, reads sum all N. Use when single-key contention is the bottleneck and the total may be eventually consistent.
Approximate distinct count (HyperLogLog) — a fixed-size probabilistic sketch (~12 KB) that counts unique items with ~2% error. Use for uniques at scale where exact membership isn't needed (unique visitors, distinct search terms).
Time-windowed (bucketed) counters — key the counter by time bucket (
```
views:{id}:2026-06-01T14
```
), increment the current bucket, sum recent buckets on read, expire old ones. Use for "last N minutes/hours" rate-style counts.
Aggregate-on-read + cached total — sum shards (or roll up) periodically and serve the cached number. Use when reads vastly outnumber writes and a slightly stale total is fine (pairs with
```
caching
```
).

单原子计数器：使用单行/单键配合原子
```
INCR
```
/
```
UPDATE +1
```
操作。适用于最热计数的峰值写入速率远低于单个节点的序列化写入吞吐量的场景。这是默认方案，不要过早过度设计。
写入分片（条带化）计数器：将一个逻辑计数拆分为N个物理分片（键格式：
```
counter:{id}:shard:{0..N-1}
```
）；每次写入随机/哈希选择一个分片进行增量，读取时求和所有N个分片。适用于单键竞争成为瓶颈，且总数可接受最终一致性的场景。
近似唯一计数（HyperLogLog）：一种固定大小的概率性数据结构（约12KB），可以约2%的误差统计唯一项目。适用于大规模唯一计数场景，且不需要精确成员列表（如独立访客、搜索关键词去重）。
时间窗口（分桶）计数器：按时间分桶设置计数器键（格式：
```
views:{id}:2026-06-01T14
```
），对当前分桶进行增量，读取时求和最近的分桶，过期旧分桶。适用于“最近N分钟/小时”的速率类统计。
读取时聚合 + 缓存总数：定期对分片求和并缓存结果，提供缓存后的数值。适用于读操作远多于写操作，且可接受轻微陈旧数据的场景（与
```
caching
```
配合使用）。

Trade-offs

权衡对比

Option	What it solves	What it worsens	Change it when
Single atomic counter	Simplest; exact; read-after-write trivial	One hot record caps write throughput; contention under spikes	Increments on one count exceed one node → shard the writes
Write-sharded counter	Spreads write load N-way; removes the hot spot	Reads cost N lookups + sum; total is eventually consistent; pick N up front	Read cost of summing N grows painful → cache the aggregate / roll up
HyperLogLog	Counts uniques in fixed tiny memory at huge scale	~2% error; can't list members or do exact counts	Exact uniques or the member set is required → use a stored set
Time-windowed buckets	Cheap rolling/rate counts; old data self-expires	More keys; window boundaries need care; cross-bucket reads sum many keys	You need an exact lifetime total → keep a separate lifetime counter
Aggregate-on-read + cache	Cheap reads of a heavy-write count	Served total lags writes by the refresh interval	Reads must be fresh-to-the-write → read shards live (eat the N-sum)

方案	解决的问题	带来的问题	何时调整
单原子计数器	实现最简单；结果精确；读写一致性易保证	单热点记录限制写入吞吐量；峰值时出现竞争	单个计数的增量超过单个节点处理能力 → 对写入进行分片
写入分片计数器	将写入负载分散N倍；消除热点	读取需要N次查询+求和；总数是最终一致性；需提前确定N值	求和N个分片的读取成本过高 → 缓存聚合结果/定期汇总
HyperLogLog	以极小的固定内存实现大规模唯一计数	约2%的误差；无法列出成员或进行精确计数	需要精确唯一计数或成员列表 → 使用存储集合
时间窗口分桶计数器	低成本实现滚动/速率统计；旧数据自动过期	需要更多键；时间窗口边界需谨慎处理；跨分桶读取需求和多个键	需要精确的生命周期总数 → 单独维护生命周期计数器
读取时聚合 + 缓存	低成本读取高写入计数的结果	返回的总数滞后于写入，延迟等于刷新间隔	读取需要与写入实时一致 → 实时读取分片求和（承担N次查询成本）

Behavior under stress

压力下的表现

A counter is a tiny thing that punches above its weight in an outage.

Hot-shard skew: if writes pick shards by
```
hash(userId)
```
instead of random, one viral actor or a bad hash can still pile onto one shard. Mitigate: pick the shard at random per write; size N to peak contention, not average.
Read amplification on spikes: when a count goes viral, reads of the total multiply the N-shard sum across the read fan-out and can overload the store. Mitigate: cache the aggregate and refresh on an interval, not per read (→
```
caching
```
).
Lost increments: fire-and-forget increments (or a crash before flush in a buffered/write-back path) silently undercount. Mitigate: use the store's atomic increment, accept the eventual-consistency window explicitly, and reconcile from a source of truth if exactness later matters.
Window-boundary stampede: time-bucketed counters all roll to a new key at the top of the hour — a synchronized cold bucket plus a flood of reads. Mitigate: pre-create buckets and jitter rollups.
Mass expiry: bucketed counters expiring together can spike the store. Mitigate: stagger TTLs.

Monitor: per-shard write distribution (skew), increments/sec vs. node ceiling, read-path latency for the N-sum, sketch error budget (HLL), and under/over-count drift against any source of truth.

计数器是一个看似微小，但在故障中影响巨大的组件。

热点分片倾斜：如果写入按
```
hash(userId)
```
而非随机选择分片，某个热门用户或糟糕的哈希算法仍可能导致负载集中在单个分片上。缓解方案：每次写入随机选择分片；根据峰值竞争而非平均负载设置N值。
峰值时的读取放大：当计数成为热点时，总数读取操作会将N分片求和的负载放大，可能导致存储系统过载。缓解方案：缓存聚合结果并按固定间隔刷新，而非每次读取都求和（参考
```
caching
```
）。
增量丢失：“一发即弃”的增量操作（或缓冲/回写路径中刷新前崩溃）会导致计数悄无声息地偏低。缓解方案：使用存储系统的原子增量操作；明确接受最终一致性窗口；若后续需要精确值，可从可信数据源进行对账。
窗口边界风暴：时间分桶计数器会在整点统一切换到新键——同步冷启动分桶加上大量读取请求会引发风暴。缓解方案：提前创建分桶，错开汇总时间。
批量过期：分桶计数器同时过期会导致存储系统负载突增。缓解方案：错开TTL时间。

监控指标：分片写入分布（倾斜情况）、每秒增量与节点上限对比、N分片求和的读取路径延迟、HyperLogLog的误差预算、与可信数据源的计数偏差。

How to apply

实施步骤

Clarify the inputs — peak increments/sec on the hottest single count, exact-vs-approximate tolerance, occurrence-vs-distinct, read rate, and freshness budget (see Clarify first). If no number shows one count is too hot, stay on a single atomic counter (YAGNI).
Pick from the trade-off table — single atomic if it fits one node; write-sharded if single-key contention is the wall; HyperLogLog for uniques; time-buckets for rolling windows. Combine (e.g. sharded + cached aggregate).
Set the key knobs — choose N (shard count) from peak contention, the shard-selection rule (random, not user-hashed), the read aggregation method, bucket granularity + TTL for windows, and the cache refresh interval.
Stress-test the choice — walk Behavior under stress: confirm shard skew, read amplification, lost increments, and window boundaries each have a mitigation the traffic profile actually needs.
Size it with numbers — N ≥ peak increments/sec ÷ per-shard write ceiling; confirm the N-sum read cost and any HLL error fit the budget (→
```
back-of-the-envelope
```
).
Pick a provider — default to the generic recipe; open a provider file only if the user named a cloud (see Choosing a provider).

明确输入条件：最热单个计数的峰值增量/秒、精确性容忍度、统计类型（出现次数/唯一项目）、读取速率、新鲜度预算（参考「先明确以下问题」）。如果没有数据表明单个计数成为瓶颈，保持使用单原子计数器（遵循YAGNI原则）。
根据权衡表选择方案：单原子计数器适用于单个节点可处理的场景；写入分片适用于单键竞争成为瓶颈的场景；HyperLogLog适用于唯一计数；时间分桶适用于滚动窗口。可组合使用（如分片+缓存聚合结果）。
设置关键参数：根据峰值竞争选择N（分片数量）、分片选择规则（随机，而非用户哈希）、读取聚合方式、时间窗口粒度+TTL、缓存刷新间隔。
压力测试所选方案：验证「压力下的表现」中的场景，确认分片倾斜、读取放大、增量丢失、窗口边界等问题都有符合流量特征的缓解方案。
量化配置：N ≥ 峰值增量/秒 ÷ 单分片写入上限；确认N分片求和的读取成本和HyperLogLog的误差在可接受范围内（参考
```
back-of-the-envelope
```
）。
选择服务商：默认使用通用方案；仅当用户指定云服务商时，查看
```
references/providers/<provider>.md
```
获取托管服务映射、配额/限制及服务商特定权衡。若该服务商无对应文件，则使用通用方案。

Dos and don'ts

注意事项

Start with a single atomic counter; shard only when a number shows one count is the bottleneck.
Pick the write shard at random so load spreads evenly regardless of the actor.
Cache the summed aggregate and refresh on an interval when reads dominate.
Use HyperLogLog for uniques-at-scale and state the ~2% error as a known cost.
Expire time-bucket keys with staggered TTLs and pre-create the next bucket.
State the eventual-consistency window out loud — sharding trades exact-now for throughput.

Don't

Don't shard a counter that a single
```
INCR
```
already handles (premature sharding adds read cost for nothing).
Don't hash the shard by user/entity ID — a hot actor reconcentrates the load.
Don't sum N shards on every read of a viral count — cache the aggregate.
Don't use a sharded/eventual counter where the number must be transactionally exact (money, sell-out inventory) →
```
consistency-coordination
```
.
Don't reach for HyperLogLog when you also need the member list or an exact count.

建议

从单原子计数器开始；仅当数据表明单个计数成为瓶颈时再进行分片。
随机选择写入分片，确保负载均匀分布，不受用户影响。
当读操作占主导时，缓存求和后的聚合结果并按固定间隔刷新。
使用HyperLogLog处理大规模唯一计数，并明确说明约2%的误差是已知成本。
为时间分桶键设置错开的TTL，并提前创建下一个分桶。
明确说明最终一致性窗口——分片方案以牺牲实时精确性换取吞吐量。

禁忌

不要对单个
```
INCR
```
已能处理的计数器进行分片——过早分片只会增加读取成本，毫无收益。
不要按用户/实体ID哈希选择分片——热门用户会重新集中负载。
不要对热点计数的每次读取都求和N个分片——缓存聚合结果。
不要在需要事务级精确计数的场景（如资金、售罄库存）使用分片/最终一致性计数器 → 参考
```
consistency-coordination
```
。
不要在需要成员列表或精确计数的场景使用HyperLogLog。

Numbers that matter

关键数值

The deciding figure is peak increments/sec on the single hottest count vs. a node's serialized write ceiling — that ratio sets N. A HyperLogLog sketch is ~12 KB for billions of uniques at ~2% standard error, regardless of cardinality — the reason it beats a stored set at scale. Reading a sharded total costs N lookups, so N trades write headroom for read cost. For QPS rates, per-node write ceilings, and storage sizing, don't restate them here — see

back-of-the-envelope

决定因素是最热单个计数的峰值增量/秒与节点序列化写入上限的比值——该比值决定N值。HyperLogLog数据结构约占12KB内存，可统计数十亿唯一项目，误差约2%，与基数无关——这是它在大规模场景下优于存储集合的原因。读取分片总数需要N次查询，因此N值是写入余量与读取成本的权衡。关于QPS速率、单节点写入上限和存储容量规划，此处不再赘述——参考

back-of-the-envelope

。

Interface sketch

接口示例

A sharded counter is a key contract, not one value:

Write:
```
INCR counter:{id}:shard:{rand(0..N-1)}
```
(atomic, fire to one shard).
Read:
```
sum(GET counter:{id}:shard:{0..N-1})
```
— or read the cached aggregate
```
count:{id}
```
refreshed every T seconds.

Distinct:

PFADD uniq:{id} {member}

then

PFCOUNT uniq:{id}

(HLL sketch).

Windowed:
```
INCR views:{id}:{bucket}
```
with a TTL; read sums the recent buckets.

Decide N, the shard-selection rule, the aggregation/refresh policy, and the window granularity up front — they are the contract, not implementation details.

分片计数器是一种键契约，而非单一值：

写入：
```
INCR counter:{id}:shard:{rand(0..N-1)}
```
（原子操作，写入单个分片）。
读取：
```
sum(GET counter:{id}:shard:{0..N-1})
```
—— 或读取缓存的聚合结果
```
count:{id}
```
，该结果每T秒刷新一次。
唯一计数：
```
PFADD uniq:{id} {member}
```
然后
```
PFCOUNT uniq:{id}
```
（HyperLogLog数据结构）。
时间窗口：
```
INCR views:{id}:{bucket}
```
并设置TTL；读取时求和最近的分桶。

需提前确定N值、分片选择规则、聚合/刷新策略、时间窗口粒度——这些是契约，而非实现细节。

Choosing a provider

选择服务商

Default to the generic recipe above. If the user names a cloud, read

references/providers/<provider>.md

for the managed-service mapping, quotas/limits, and provider-specific trade-offs. If no file exists for that provider, the generic recipe is the answer.

默认使用上述通用方案。若用户指定云服务商，查看

references/providers/<provider>.md

获取托管服务映射、配额/限制及服务商特定陷阱。若该服务商无对应文件，则通用方案即为答案。

Diagram

示意图

To visualize the fan-out write path (writer → random shard) and the aggregate-on-read sum (read → N shards → cached total), use the in-plugin

architecture-diagram

skill — shards share the store color, the read fan-out is a dashed sum arrow, and the cached aggregate sits in the cache color.

如需可视化扇出写入路径（写入者→随机分片）和读取时聚合求和（读取→N个分片→缓存总数），可使用插件内的

architecture-diagram

技能——分片使用存储系统颜色，读取扇出用虚线求和箭头，缓存聚合结果使用缓存颜色。

Related building blocks

References

参考资料

references/deep-dive.md
— shard-count math, random vs. hashed selection, HyperLogLog mechanics and error, time-bucket layout, roll-up/aggregation patterns, and reconciliation. Read when designing the counter in detail.
references/providers/{generic,aws,azure,gcp}.md
— service mappings, atomicity/contention limits, and pitfalls per environment.

references/deep-dive.md
—— 分片数量计算、随机vs哈希选择、HyperLogLog机制与误差、时间分桶布局、汇总/聚合模式及对账细节。设计计数器细节时可阅读。
references/providers/{generic,aws,azure,gcp}.md
—— 各环境下的服务映射、原子性/竞争限制及陷阱。