api-idempotency-retry

Compare original and translation side by side

🇺🇸

Original

English

🇨🇳

Translation

Chinese

API Idempotency and Retry

API幂等性与重试

You design the two halves of one guarantee: server-side idempotency keys that make a retried write safe, and client-side retry behavior that uses them responsibly. Shipping either half alone is the trap - Resilience4j's own docs state it as a hard constraint, not a suggestion: retried operations must be idempotent, so retry guidance without key support (or the reverse) leaves the guarantee open exactly where a dropped connection tests it.

Two facts frame everything below:

Exactly-once delivery is impossible over an unreliable network (Tyler Treat's canonical argument, grounded in the FLP result). What a key actually buys is effectively-once: at-least-once delivery plus duplicate collapse, bounded to the scope where idempotency is actually implemented.
There is no ratified standard behind the
```
Idempotency-Key
```
header. The IETF draft (
```
draft-ietf-httpapi-idempotency-key-header
```
) expired in April 2026 without ratification. The header name is consistent across the industry because Stripe popularized it, but the semantics (TTL, mismatch behavior, concurrency handling) are whatever each provider decided, so yours must be designed and documented, never assumed.

你设计同一个保证的两个半部分：服务端幂等键让重试的写入安全，客户端重试行为负责合理地使用它们。只交付其中一半就是陷阱——Resilience4j 的文档将其表述为硬性约束而非建议：被重试的操作必须是幂等的，因此没有键支持的重试指南（或反之）会在断连正好考验它的地方留下保证缺口。

以下所有内容由两个事实框定：

在不可靠网络上，精确一次（exactly-once）投递是不可能的（Tyler Treat 的经典论证，基于 FLP 结果）。键实际上买到的是_有效一次（effectively-once）_：至少一次投递加上重复折叠，限定在幂等实际实现的作用域内。
```
Idempotency-Key
```
头背后没有批准的标准。IETF 草案（
```
draft-ietf-httpapi-idempotency-key-header
```
）于2026年4月过期未获批准。头名称在整个行业一致是因为 Stripe 推广了它，但语义（TTL、不匹配行为、并发处理）由各家自己决定，所以你的设计必须被设计和记录，而不能假设。

Clarifying questions

澄清问题

Ask these before designing anything; each answer changes a later step. Batch them - this is a tactical design task, not a strategy interview.

Paradigm: REST, gRPC, or both? (gRPC carries deadline propagation natively; REST needs the header convention and explicit deadline plumbing)
Which endpoints are non-idempotent? Request the endpoint list or OpenAPI spec and enumerate every state-changing operation.
What storage backs the key store - a relational DB with unique constraints and transactions, a conditional-write KV store, or only a cache? (rules atomic-claim options in or out - see step 4)
Do any requests call an external, non-rollbackable system mid-flight (a charge, an outbound email)? (drives recovery points - see step 5)
Which SDK languages do you ship, and do integrators also call raw HTTP? (drives how much of the retry mechanics you can default for them - see step 6)
What is the longest possible retry chain - dead-letter-queue replay windows, dispute windows, scheduled re-runs? (drives TTL - see step 3)
By when must the guarantee hold: a one-off fix (stop the duplicate charges one client is filing tickets about), or a compounding contract every future endpoint inherits? What is the effort ceiling on the claim mechanism - a ship date that rules out a schema or isolation-level migration, or existing fluency with SERIALIZABLE workloads? (re-ranks the claim menu - see step 4: a hard date promotes whatever the current store already supports, a compounding mandate promotes the richer claim lifecycle, and a ceiling that forbids a schema migration deletes the rungs it cannot reach rather than ranking them last)

在设计之前先问这些；每个答案都会改变后续步骤。批量提问——这是一个战术性设计任务，不是策略面试。

范式：REST、gRPC，还是两者？（gRPC 原生携带截止时间传递；REST 需要头约定和显式截止时间管道）
哪些端点不是幂等的？请求端点列表或 OpenAPI 规范，枚举每个改变状态的操作。
什么存储支撑键存储——具有唯一约束和事务的关系型数据库、条件写入 KV 存储，还是只有缓存？（决定原子声明的选项——见步骤4）
是否有任何请求在途中调用外部不可回滚系统（一笔扣款、一封外发邮件）？（驱动恢复点——见步骤5）
你发布哪些 SDK 语言，集成方是否也直接调用原始 HTTP？（决定你能为他们默认多少重试机制——见步骤6）
最长的可能重试链是什么——死信队列重放窗口、争议窗口、计划重跑？（驱动 TTL——见步骤3）
保证必须到何时生效：一次性修复（停止一个客户端正在提交工单的重复扣费），还是每个未来端点都继承的复合契约？声明机制的工作量上限是什么——排除 schema 或隔离级别迁移的发布日期，还是对 SERIALIZABLE 工作负载已有流畅经验？（重新排列声明菜单——见步骤4：硬性日期会把当前存储已支持的方式提前，复合任务会推动更丰富的声明生命周期，而禁止 schema 迁移的上限会删除它够不到的梯级而不是排在最后）

Who controls the retry loop

谁控制重试循环

Every retry decision is made by a developer or a program, whoever bought the product. The design splits by who controls the retry loop:

First-party developers - you control both ends; internal convention plus code review covers most of it. The cheapest audience to serve.
Third-party integrators calling raw HTTP - they write their own retry loop from your docs alone. They need the full contract published: key semantics, TTL, mismatch and in-flight behavior, and a worked backoff example, because no standard fills the gaps for them.
SDK-mediated callers - your SDK is their retry loop. Its defaults (backoff, jitter, attempt caps, key auto-generation) are the contract most callers will never read past, so wrong defaults ship duplicates at fleet scale.

Design for raw-HTTP integrators first - a contract complete enough for them is what your own SDKs then automate - and treat SDK defaults as the highest-leverage artifact, since they decide behavior for the majority who never read the docs.

每个重试决策由开发者或程序做出，无论是谁购买了产品。设计按谁控制重试循环来划分：

第一方开发者——你控制两端；内部约定加代码评审覆盖大部分。最便宜的受众。
第三方集成方直接调用原始 HTTP——他们仅根据你的文档自己写重试循环。他们需要完整发布的契约：键语义、TTL、不匹配和并发行为，以及一个完整的退避示例，因为没有标准为他们填补空白。
SDK 中介的调用者——你的 SDK _就是_他们的重试循环。它的默认值（退避、抖动、尝试上限、键自动生成）是大多数调用者永远不会读更多的契约，所以错误的默认值会在整个集群规模上制造重复。

先为原始 HTTP 集成方设计——一个足够完整的契约正是你自己的 SDK 随后自动化的东西——并把 SDK 默认值视为最高杠杆的产物，因为它们决定了大多数从不读文档的人的行为。

Workflow

工作流

Identify retry-unsafe endpoints.
Design the key contract.
Set storage and TTL.
Claim the key atomically.
Design the response semantics.
Write client retry guidance and SDK defaults.
Document the contract for integrators.

Each step has a section below, in order.

识别不适合重试的端点。
设计键契约。
设置存储和 TTL。
原子地声明键。
设计响应语义。
编写客户端重试指南和 SDK 默认值。
为集成方记录契约。

每个步骤下面有一节，按顺序。

1. Identify retry-unsafe endpoints

1. 识别不适合重试的端点

RFC 9110 - the one ratified document relevant here - defines GET, PUT, and DELETE as idempotent methods and POST as not. It says nothing about the
```
Idempotency-Key
```
retrofit pattern.
Enumerate every state-changing endpoint. Each must either accept an idempotency key or be explicitly documented as unsafe to retry - the review-level check
```
samber/developer-platform-skills@public-api-design-review
```
performs; this skill builds what makes the first option true.
Verify claimed idempotency, don't trust the method: a PUT that appends to a log or a DELETE that decrements a counter is not idempotent whatever the method promises.

RFC 9110——这里唯一相关的已批准文档——将 GET、PUT 和 DELETE 定义为幂等方法，POST 不是。它对
```
Idempotency-Key
```
改造模式只字未提。
枚举每个改变状态的端点。每个要么接受幂等键，要么被明确记录为不可安全重试——这是
```
samber/developer-platform-skills@public-api-design-review
```
执行的评审级别检查；本 skill 构建让第一个选项成立的东西。
验证声称的幂等性，不要相信方法：一个追加到日志的 PUT 或递减计数器的 DELETE 无论方法承诺什么都不是幂等的。

2. Design the key contract

2. 设计键契约

The client mints the key, the server never derives it. AWS's Builders' Library names the rejected alternative: a server-side hash of request parameters breaks when two genuinely identical requests are both intended (launching two identical instances on purpose) - a caller-provided key expresses intent an inferred one cannot. Stripe, AWS, Twilio, and Resend all put the client in charge.
Scope keys per caller -
```
(account_id, idempotency_key)
```
- never globally. Stripe, AWS, and the standard Postgres reference implementation all key on the pair, so the same key string reused by different accounts never collides.
Recommend a derivation to clients. Three work; rank them for the client rather than listing them (Resend's own recommendation order):
- value:
```
event-derived > request-scoped > random UUID reused across retries
```
- effort:
```
event-derived > request-scoped > random UUID reused across retries
```
- efficiency:
```
event-derived > request-scoped > random UUID reused across retries
```
  - value and effort rank identically here, so their spread decides the ratio: minutes wide on effort (name the key after an identifier the caller already holds) against the whole guarantee on value.
- Default rung: event-derived (
```
order-confirm-${orderId}
```
  ) - deterministic by construction, so a retry cannot fail to reuse the key. Move down to request-scoped when no stable business identifier exists at call time, and to a reused random UUID when the caller is a stateless script. Nothing is starved here: every rung costs one line of client code.
- Never rank a fourth rung below these: a fresh value per attempt (
```
Date.now()
```
  ,
```
randomUUID()
```
  regenerated on retry) defeats the mechanism entirely, so document it as the anti-pattern instead.
Guard the payload: the same key with a different request body is a client bug - reject it (409 is the field convention; Resend names it
```
409 invalid_idempotent_request
```
), never silently replay. The expired IETF draft generalizes this into an optional payload fingerprint stored beside the key; a checksum of the payload is the simplest form.
Store both outcomes. Stripe saves failures as well as successes - including 500s - so a retry replays the exact original result. Decide explicitly whether a stored failure replays verbatim (Stripe v1) or the operation may be re-attempted without side effects (Stripe v2); leaving it undecided is a design gap.

客户端铸造键，服务端从不推导。AWS Builders' Library 点名被否定的替代方案：服务端对请求参数的哈希在两个真正相同的请求都被有意发送时失效（故意启动两个相同实例）——调用方提供的键表达了推断无法表达的意图。Stripe、AWS、Twilio 和 Resend 都让客户端负责。
按调用方隔离键——
```
(account_id, idempotency_key)
```
——绝不全局。Stripe、AWS 和标准 Postgres 参考实现都以该二元组为键，因此不同账户复用相同键字符串永远不会冲突。
向客户端推荐派生方式。三个可行；为客户端排序而不是罗列（Resend 自己的推荐顺序）：
- value：
```
event-derived > request-scoped > random UUID reused across retries
```
- effort：
```
event-derived > request-scoped > random UUID reused across retries
```
- efficiency：
```
event-derived > request-scoped > random UUID reused across retries
```
  ——价值和努力在这里排名相同，所以它们的差距决定比例：努力上几分钟（以调用者已持有的标识符命名键）对比价值上的整个保证。
- 默认梯级：事件派生（
```
order-confirm-${orderId}
```
  ）——构造上就是确定性的，所以重试不可能无法复用键。当调用时没有稳定的业务标识符时降到请求作用域，当调用者是无状态脚本时降到复用的随机 UUID。这里没有挨饿：每个梯级只花一行客户端代码。
- 永远不要把第四个梯级排在这些下面：每次尝试生成新值（
```
Date.now()
```
  、重试时重新生成的
```
randomUUID()
```
  ）完全破坏了机制，所以把它记录为反模式。
保护载荷：相同键不同请求体是客户端 bug——拒绝它（409 是领域约定；Resend 称之为
```
409 invalid_idempotent_request
```
），绝不静默重放。过期的 IETF 草案将其泛化为存储在键旁边的可选载荷_指纹_；载荷的校验和是最简单的形式。
存储两种结果。Stripe 保存失败也保存成功——包括 500——因此重试精确重放原始结果。明确决定存储的失败是原样重放（Stripe v1）还是操作可以在没有副作用的情况下重新尝试（Stripe v2）；悬而未决就是设计缺口。

3. Set storage and TTL

3. 设置存储和 TTL

Retention is two separate concerns, not one TTL (Brandur Leach's reference implementation):
- Replay window (~24h) - governs idempotent-replay correctness; also the documented convention at Stripe, Resend, and Twilio.
- Reaper threshold (~72h) - a longer operational window that keeps records alive for human recovery, so a bad Friday deploy's failed requests survive the weekend for Monday's fix.
Set the replay window from the longest retry chain (question 6), never from storage cost. A 24-hour TTL behind a 7-day DLQ replay is a duplicate waiting to happen. AWS retains EC2 run tokens for the resource's lifetime plus a grace period - which is what lets a late retry get a consistent answer even after the resource was deleted.
After expiry, treat a reused key as a brand-new request - every provider surveyed does; none returns an error for post-TTL reuse.
Store the full response verbatim (status, headers that matter, body), not a summary - the replay must be indistinguishable from the original.

保留是两个独立关注点，不是一个 TTL（Brandur Leach 的参考实现）：
- 重放窗口（约24h）——管理幂等重放正确性；也是 Stripe、Resend 和 Twilio 的记录约定。
- 收割阈值（约72h）——更长的操作窗口，让记录为人工恢复而存活，所以糟糕的周五部署的失败请求能挺过周末到周一的修复。
根据最长的重试链设置重放窗口（问题6），绝不根据存储成本。7天 DLQ 重放后面的24小时 TTL 是等待发生的重复。AWS 将 EC2 运行令牌保留到资源生命周期外加宽限期——这让延迟的重试即使资源已删除也能得到一致的答案。
过期后，将复用的键视为全新请求——调查的每个提供商都如此；没有谁对 TTL 后复用返回错误。
原样存储完整响应（状态、重要的头、正文），不是摘要——重放必须与原始响应无法区分。

4. Claim the key atomically

4. 原子地声明键

A check-then-insert is a TOCTOU race under concurrent retries: both callers see "absent" and both proceed. The claim must be a single atomic operation the storage layer arbitrates. Three mechanisms appear in production, ranked:

efficiency:

DB unique constraint > conditional-write OCC > row lock + SERIALIZABLE

effort:

row lock + SERIALIZABLE > conditional-write OCC > DB unique constraint

value:

row lock + SERIALIZABLE > DB unique constraint == conditional-write OCC

Default rung: a unique constraint on
(account_id, idempotency_key)
. The database picks the single winner; the loser gets its 409 from the
```
INSERT
```
failure itself, not an application-level check. Nearly free - one index - and transaction isolation gives the fencing guarantee below at no extra cost.
Conditional-write OCC (DynamoDB-style) delivers the same single-winner claim on stores without unique-constraint transactions - the tie on the value line is real: same guarantee, different storage family. Your answer to question 3 rules one of these two out rather than ranking it; a ruled-out option is deleted, not demoted.
Row lock + SERIALIZABLE transaction is the starved option.
- Value: a
```
locked_at
```
  column inside a SERIALIZABLE transaction represents in-flight as a first-class state, survives lock expiry, and anchors resumable recovery points.
- Effort: isolation-level tuning, serialization-abort retry handling - so it loses every efficiency round.
- Promote it when requests span foreign state mutations (question 4): multi-step requests need the richer claim lifecycle, not just a winner-picker.
Whatever the mechanism: the claim and the business mutation must commit as one ACID unit - AWS states this as a hard requirement, not a best practice. A claim outside the transaction it guards can record a token for a resource that was never created, or the reverse.
A bare Redis
```
SETNX
```
+ TTL lock is not a fourth option. Kleppmann's analysis: without a fencing token, a client paused past its lease expiry can still write after the lock moved on - no timing assumption saves you. No surveyed vendor uses a distributed lock for the claim; if Redis is all you have, layer fencing tokens on top, and know the DB constraint sidesteps the problem entirely.
This ranking is a default, not a law. Re-rank against the storage answer and the effort ceiling (questions 3 and 7): a team already running SERIALIZABLE workloads gets the starved option far cheaper than the effort line assumes. A ship date that forbids a schema migration promotes whichever mechanism their current store already supports.

检查后插入是并发重试下的 TOCTOU 竞争：两个调用者都看到“不存在”然后都继续。声明必须是存储层仲裁的单个原子操作。生产中出现的三种机制，排序：

efficiency：

DB unique constraint > conditional-write OCC > row lock + SERIALIZABLE

effort：

row lock + SERIALIZABLE > conditional-write OCC > DB unique constraint

value：

row lock + SERIALIZABLE > DB unique constraint == conditional-write OCC

默认梯级：对
(account_id, idempotency_key)
的唯一约束。数据库选出唯一赢家；失败者从
```
INSERT
```
失败本身得到它的 409，而不是应用层检查。几乎免费——一个索引——事务隔离以零额外成本提供下面的 fencing 保证。
条件写 OCC（DynamoDB 风格） 在没有唯一约束事务的存储上提供同样的单赢家声明——价值线上的平局是真实的：同样的保证，不同的存储家族。你对问题3的回答排除这两者之一而不是排序；被排除的选项被删除，不是降级。
行锁 + SERIALIZABLE 事务是被饿死的选项。
- 价值：SERIALIZABLE 事务中的
```
locked_at
```
  列把_进行中_表示为头等状态，在锁过期后存活，并锚定可恢复的恢复点。
- 努力：隔离级别调优、序列化中止重试处理——所以它在每个效率轮都输。
- 当请求跨外部状态变更（问题4）时提升它：多步请求需要更丰富的声明生命周期，而不仅仅是赢家选择器。
无论什么机制：声明和业务变更必须作为一个 ACID 单元提交——AWS 将其表述为硬性要求，不是最佳实践。事务外的声明可以为从未创建的资源记录令牌，或者反过来。
裸 Redis
```
SETNX
```
+ TTL 锁不是第四个选项。Kleppmann 的分析：没有 fencing 令牌，超过租约过期被暂停的客户端仍然可以在锁移走后写入——没有任何时间假设能救你。调查的供应商都没有用分布式锁做声明；如果只有 Redis，在上面分层 fencing 令牌，并知道数据库约束完全绕过了问题。
这个排序是默认，不是法律。根据存储答案和工作量上限（问题3和7）重新排序：已经在运行 SERIALIZABLE 工作负载的团队得到被饿死的选项比努力线假设的便宜得多。禁止 schema 迁移的发布日期会提升他们当前存储已支持的任何机制。

5. Design the response semantics

5. 设计响应语义

The bug in most homegrown implementations is treating a key as binary - seen or not-seen. A key is a three-state machine: absent, in-flight, complete. The retry that hits an in-flight key is not the bug ("your client library did exactly what it was configured to do when the connection dropped"); the bug is a server with no way to say "seen, not finished."

Outcome	What happened	Correct behavior
Never received	Network drop before processing	Retry proceeds as a first attempt; the claim wins normally
Received, not processed	Claim held, no terminal result yet	Return an in-flight conflict, or resume from the last recovery point once the original lock expired
Processed, response lost	Result committed; response never arrived	Replay the stored response verbatim - no re-execution

For the in-flight case, pick one:
- Reject with 409 (Resend's
```
409 concurrent_idempotent_requests
```
  , documented advice "wait and retry") - simplest and safest default.
- Block until the bounded result, when the caller needs it synchronously.
- Return 202 plus a status URL, for long-running effects.
Never let the second caller through because the first "seems stuck." The expired IETF draft deliberately made in-flight conflict and completed replay different responses, not fallbacks of each other.
For multi-step requests (question 4), decompose into atomic phases separated by each foreign state mutation, persisting a recovery point between phases - a retried request resumes exactly where it left off instead of re-running an external call already made. Record intent before calling out, so a crash between call and response leaves evidence.
Run recovery as active background jobs, not passive expiry (Brandur's model):
- A completer pushes abandoned in-flight requests through to completion.
- A reaper deletes expired keys and logs what it couldn't finish for a human.
What the error responses look like - codes, envelope,
```
retryable
```
flag,
```
Retry-After
```
- is
```
samber/developer-platform-skills@api-error-design
```
's territory; consume its signals, don't redesign them here. Distinguishing your two 409 meanings (payload mismatch vs in-flight) with distinct machine-readable codes is that skill's taxonomy applied to this skill's states.

See references/key-lifecycle-walkthrough.md for the full worked lifecycle: schema, claim query, all three states' responses, and a multi-step recovery-point example.

大多数自制实现中的 bug 是把键当作二元的——见过或没见过。键是三态机：不存在、进行中、完成。命中进行中键的重试不是 bug（“你的客户端库在连接断开时完全按配置行事”）；bug 是一个服务端没有办法说“见过，但没完成”。

结果	发生了什么	正确行为
从未收到	处理前网络断开	重试作为第一次尝试进行；声明正常获胜
收到，未处理	声明已持有，还没有最终结果	返回进行中冲突，或等原始锁过期后从最后恢复点恢复
已处理，响应丢失	结果已提交；响应从未到达	原样重放存储的响应——不重新执行

对进行中情况，选一个：
- 用 409 拒绝（Resend 的
```
409 concurrent_idempotent_requests
```
  ，文档建议“等待并重试”）——最简单最安全的默认。
- 当调用者需要同步结果时，阻塞直到有界结果。
- 对长时间运行的效果，返回 202 加状态 URL。
绝不要让第二个调用者因为第一个“看起来卡住”而通过。过期的 IETF 草案故意让进行中冲突和完成重放是_不同_的响应，而不是彼此的 fallback。
对多步请求（问题4），分解为每个外部状态变更分隔的原子阶段，在阶段之间持久化恢复点——重试的请求从它停下的地方精确恢复，而不是重新运行已经发出的外部调用。在调用外发_之前_记录意图，这样调用和响应之间的崩溃留下证据。
将恢复作为主动后台任务运行，而不是被动过期（Brandur 模型）：
- completer 把被遗弃的进行中请求推到完成。
- reaper 删除过期键并记录它无法完成的内容给人类处理。
错误响应长什么样——代码、信封、
```
retryable
```
标志、
```
Retry-After
```
——是
```
samber/developer-platform-skills@api-error-design
```
的领地；消费它的信号，不要在这里重新设计。用不同的机器可读代码区分你的两个 409 含义（载荷不匹配 vs 进行中）是该 skill 的分类法应用到这个 skill 的状态上。

完整的生命周期示例见 references/key-lifecycle-walkthrough.md：schema、声明查询、所有三态响应，以及一个多步恢复点示例。

6. Write client retry guidance and SDK defaults

6. 编写客户端重试指南和 SDK 默认值

Retry an explicit allowlist, never a blanket catch: 429 (honoring
```
Retry-After
```
), 502, 503, 504, connection timeouts and resets - 500 cautiously (some 500s are a real bug that retries identically). Never retry non-429 4xx, auth failures, or exceptions that are bugs. Four independent sources (tenacity, Resilience4j, Resend, and the response-signaling side) converge on this exact split - cite it as convergent practice, not one vendor's opinion.
Never retry a read timeout on a non-idempotent operation without an idempotency key already in place. This is the exact boundary condition the server half of this skill exists to close.
Backoff must carry jitter. Ranked (Marc Brooker's AWS analysis, the canonical source):
- value:
```
full jitter > decorrelated jitter > equal jitter > no jitter
```
- effort:
```
full jitter == decorrelated jitter == equal jitter == no jitter
```
  - a real tie, not a dodge: each is one formula or one library flag, identical to write, test, and reverse, so no rung buys its value with more of the reader's time.
- efficiency: the value order unchanged, since equal effort cancels out of the ratio.
- Default rung: full jitter (
```
sleep = random(0, min(cap, base * 2^attempt))
```
  ) - lowest total client work and lowest server load, at slightly more total time to completion.
- Starved by that order: decorrelated jitter, which completes faster for more total work. Promote it when a user-facing call sits inside a tight deadline and time-to-success outweighs fleet load. Equal jitter is Brooker's "clear loser" among jittered options, and no jitter re-synchronizes the fleet into the exact spike that caused the failures - keep neither in the menu you hand an integrator.
- Re-rank against what you know about the caller: a single-instance batch job pays none of the fleet synchronization cost this order assumes.
Bound both attempt count and total duration, not attempts alone - a per-attempt-reasonable loop can blow past the caller's own budget. Log every retry (attempt, error, next wait); a silent retry loop hides an outage in the making.
Add a fleet-wide retry budget, distinct from per-caller caps: stop retrying once retries exceed a set share of traffic over a sliding window. A hundred instances each retrying three times amplify load 300x while every instance looks reasonable in isolation. AWS's token bucket is the named implementation, tuned to engage the quota sooner in a real outage: 14 tokens per transient-error retry, 5 per throttle retry.
Propagate the deadline: one overall deadline set at the top, per-attempt timeouts budgeted inside it - never a fresh overall timeout per attempt, which silently extends the caller's real wait. Set single timeouts at 2-3x measured p99, not an OS default. gRPC's relative-timeout-on-the-wire model is the citable mechanics for chains.
Point integrators at their language's retry library (tenacity, Resilience4j, or equivalent) before describing formulas - the formula is a one-liner there, pre-debugged. When pairing with a circuit breaker, state the stacking order:
```
Retry → CircuitBreaker → RateLimiter → Bulkhead
```
.
For SDKs you ship: default retries ON with backoff + jitter and a bounded attempt count (AWS defaults to 3 total), and auto-generate idempotency keys for unsafe operations where the SDK can. Defaults are the contract for callers who never read docs. How your retry defaults interact with the platform's rate-limit policy - whether SDK retries count against quota, adaptive throttling - is
```
samber/developer-platform-skills@api-rate-limit-policy
```
territory.

Formulas, budget mechanics, and deadline math: references/backoff-jitter-formulas.md. Cross-vendor defaults to calibrate yours against: references/vendor-sdk-retry-defaults.md.

重试显式白名单，而不是一刀切 catch：429（尊重
```
Retry-After
```
）、502、503、504、连接超时和重置——500 谨慎（有些 500 是真正 bug，重试也一样失败）。绝不重试非 429 的 4xx、认证失败或属于 bug 的异常。四个独立来源（tenacity、Resilience4j、Resend 和响应信号方）在这个确切划分上收敛——引用为趋同实践，不是某一家供应商的观点。
在没有幂等键已经就位的情况下，绝不对非幂等操作重试读超时。这正是本 skill 服务端一半存在要闭合的边界条件。
退避必须带抖动。排序（Marc Brooker 的 AWS 分析，权威来源）：
- value：
```
full jitter > decorrelated jitter > equal jitter > no jitter
```
- effort：
```
full jitter == decorrelated jitter == equal jitter == no jitter
```
  ——真正的平局，不是回避：每个都是一个公式或一个库标志，写、测试和回滚都一样，所以没有梯级用更多读者时间来买它的价值。
- efficiency：价值顺序不变，因为相等的努力在比例中抵消。
- 默认梯级：full jitter（
```
sleep = random(0, min(cap, base * 2^attempt))
```
  ）——客户端总工作量最低，服务端负载最低，完成总时间略长。
- 被该顺序饿死的是：decorrelated jitter，它完成更快但总工作量更多。当面向用户的调用在严格截止时间内且成功时间比集群负载更重要时提升它。Equal jitter 是 Brooker 眼中抖动选项中的“明显输家”，而 no jitter 会把集群重新同步成导致失败的完全相同尖峰——两者都不要留在你交给集成方的菜单里。
- 根据你对调用者的了解重新排序：单实例批处理作业不承担该顺序假设的集群同步成本。
同时限制尝试次数和总时长，而不只是尝试次数——每次尝试看起来合理的循环可能超出调用者自己的预算。记录每次重试（尝试、错误、下次等待）；静默重试循环会隐藏正在形成的故障。
增加一个集群范围内的重试预算，区别于每个调用者的上限：一旦重试在滑动窗口内超过流量的一定份额就停止重试。一百个实例每个重试三次会在每个实例单独看来都合理的情况下放大 300 倍负载。AWS 的令牌桶是被点名的实现，调优为在真实故障中更早启动配额：每次瞬态错误重试 14 个令牌，每次限流重试 5 个。
传递截止时间：一个总截止时间在顶层设置，每次尝试的超时在它内部做预算——绝不要每次尝试都设置新的总体超时，这会静默延长调用者真实等待。单个超时设为实测 p99 的 2-3 倍，而不是操作系统默认。gRPC 的线上相对超时模型是链条的可引用机制。
在描述公式之前把集成方指向他们语言的重试库（tenacity、Resilience4j 或等价物）——公式在那里是一行，已经调试过。与熔断器搭配时，说明堆叠顺序：
```
Retry → CircuitBreaker → RateLimiter → Bulkhead
```
。
对你发布的 SDK：默认开启重试，带退避 + 抖动和有界尝试次数（AWS 默认共 3 次），并在 SDK 可以为不安全操作自动生成幂等键时自动生成。默认值是那些从不读文档的调用者的契约。你的重试默认值如何与平台限流策略交互——SDK 重试是否计入配额、自适应限流——是
```
samber/developer-platform-skills@api-rate-limit-policy
```
的领地。

公式、预算机制和截止时间数学：references/backoff-jitter-formulas.md。用于校准你的跨厂商默认值：references/vendor-sdk-retry-defaults.md。

7. Document the contract for integrators

7. 为集成方记录契约

Because no standard defines the semantics, your documentation is the only normative source your integrators have. Document, per the audience split above:

Which endpoints accept the key, the header name, max key length, and the recommended derivation.
Scoping, the replay window as a concrete number, and what happens after it expires ("treated as a new request").
Both conflict behaviors, each with its distinct error code: payload mismatch and in-flight duplicate.
Whether a stored failure replays or re-attempts (the Stripe v1/v2 distinction, applied to your API).
A worked retry loop with concrete numbers - schedule, cap, jitter - like Resend's documented
```
1s → 2s → 4s → 8s
```
capped at 30s.
Never write "exactly-once delivery" - the claim is provably false across an open network. Write what is true instead: retries are safe within the documented key scope and window. Correct the claim wherever marketing already made it (Treat's steelman: Kafka's exactly-once holds only inside a closed transactional system; a public API is by definition not one).

因为没有标准定义语义，你的文档是集成方拥有的唯一规范来源。按照上面的受众划分记录：

哪些端点接受该键、头名称、最大键长度和推荐派生方式。
作用域、作为具体数字的重放窗口，以及过期后发生什么（“视为新请求”）。
两个冲突行为，每个有独立错误码：载荷不匹配和进行中重复。
存储的失败是重放还是重新尝试（Stripe v1/v2 区别，应用到你的 API）。
一个带具体数字的完整重试循环——计划、上限、抖动——比如 Resend 记录的
```
1s → 2s → 4s → 8s
```
上限 30s。
永远不要写“精确一次投递”——这个声称在开放网络上可证明是假的。写真实的内容：在文档化的键作用域和窗口内重试是安全的。无论营销已经在哪儿这么说了，纠正它（Treat 的稻草人：Kafka 的 exactly-once 只在封闭事务系统内成立；公共 API 按定义不是）。

Failure modes

失败模式

Anti-pattern checklist - each is a direct audit finding:

A key accepted but never stored or honored - tells the client retrying is safe when it isn't; worse than no key at all.
A fresh key value generated per attempt.
Check-then-insert claiming (TOCTOU race).
Binary key state with no in-flight representation.
Same key, different payload, silently replayed.
Replay window shorter than the longest retry chain (the 24h-TTL-behind-7-day-DLQ trap).
A bare
```
SETNX
```
lock with no fencing token as the claim.
No decision on whether stored failures replay or re-execute.
Retry loop that resets its overall timeout each attempt.
Backoff without jitter.
"Exactly-once" anywhere in the docs.

反模式清单——每项都是直接的审计发现：

键被接受但从未存储或兑现——告诉客户端重试安全，其实不安全；比没有键更糟。
每次尝试生成新的键值。
检查后插入声明（TOCTOU 竞争）。
二元键状态，没有进行中表示。
相同键、不同载荷，静默重放。
重放窗口短于最长重试链（24h-TTL-在-7天-DLQ-后面的陷阱）。
裸
```
SETNX
```
锁没有 fencing 令牌作为声明。
没有决定存储的失败是重放还是重新执行。
每次尝试重置总体超时的重试循环。
没有抖动的退避。
文档中任何地方的“精确一次”。

Measurement

度量

Concurrency gate: fire N identical concurrent requests with one key against a state-changing endpoint - exactly one side effect must execute; the other N-1 get the in-flight conflict or the stored replay. Binary pass; iterate the claim design until it holds. This is the definitional guarantee, testable directly.
Crash-recovery gate: kill the server between atomic phases of a multi-step request, then retry - the request must resume from its recovery point with zero repeated external calls. Binary pass where question 4 answered yes.
Duplicate side-effect rate in production: within the documented scope and window the design target is zero by definition - track it as the alarm that the gates regressed, not as a tunable threshold.
Orphaned-key rate (claimed, never completed - the completer's queue): no published industry threshold exists, so set the baseline from your own first month and watch the trend, as with any unsourced operational metric.

并发门：对一个改变状态的端点用同一个键发起 N 个相同并发请求——必须恰好执行一个副作用；其他 N-1 个得到进行中冲突或存储的重放。二元通过；迭代声明设计直到成立。这是定义性保证，可直接测试。
崩溃恢复门：在多步请求的原子阶段之间杀死服务器，然后重试——请求必须从它的恢复点恢复，零重复外部调用。在问题4回答是的地方二元通过。
生产中的重复副作用率：在文档化作用域和窗口内，设计目标按定义是零——把它作为门回归的警报跟踪，而不是可调阈值。
孤儿键率（已声明、从未完成——completer 的队列）：没有已发布的行业阈值，所以像任何无来源的运营指标一样，用你自己第一个月设定基线并观察趋势。

Invocation examples

调用示例

"Add Idempotency-Key support to our POST /payments and POST /orders endpoints - we're seeing duplicate charges when mobile clients retry on timeout."
"Write the retry guidance section of our API docs, and pick the backoff defaults for the Python and Node SDKs we ship."
"Two concurrent webhook handlers both created the same invoice - audit our idempotency implementation and find the race."

“给我们的 POST /payments 和 POST /orders 端点添加 Idempotency-Key 支持——当移动客户端超时重试时，我们看到重复扣费。”
“编写我们 API 文档的重试指南部分，并为我们发布的 Python 和 Node SDK 选择退避默认值。”
“两个并发 webhook 处理程序都创建了相同的发票——审计我们的幂等实现并找到竞争。”

References

参考

references/key-lifecycle-walkthrough.md - worked key lifecycle: schema, atomic claim, three-state responses with real 409 codes, recovery points, completer/reaper.
references/backoff-jitter-formulas.md - the three jitter formulas with trade-offs, retry-budget mechanics, deadline propagation, timeout setting.
references/vendor-sdk-retry-defaults.md - how AWS, Stripe, Google Cloud, Twilio, and Resend default retry and idempotency behavior; what to copy and what to flag.

references/key-lifecycle-walkthrough.md——完整键生命周期：schema、原子声明、带真实 409 代码的三态响应、恢复点、completer/reaper。
references/backoff-jitter-formulas.md——三种抖动公式及权衡、重试预算机制、截止时间传递、超时设置。
references/vendor-sdk-retry-defaults.md——AWS、Stripe、Google Cloud、Twilio 和 Resend 如何默认重试和幂等行为；复制什么，标记什么。