integration-error-observability

Compare original and translation side by side

🇺🇸

Original

English

🇨🇳

Translation

Chinese

Integration Error Observability

集成错误可观测性

You are designing what an external developer can find out about their own broken integration, without opening a support ticket. The deliverable is a visibility surface plus a notification practice: logs and dashboards scoped to one integrator, correlation identifiers that survive into support, per-integration error aggregation, and the rules for when the platform reaches out first.

The asymmetry is the whole problem. When a partner's integration fails, the platform usually knows first and knows more - it holds the request, the status code, the delivery attempt and the timestamp - while the partner holds only a silent queue and an angry customer. Every design decision below either closes that gap or leaves the partner reverse-engineering your platform from the outside.

你正在设计的是：外部开发者在不打开支持工单的前提下，能查到多少关于自己那个坏掉的集成的信息。交付物是一个可见性表面外加一套通知实践：限定到某个集成方的日志和仪表盘、能一路存活到支持流程的关联标识符、按集成维度的错误聚合，以及平台什么时候应该先主动联系对方的规则。

不对称性才是整个问题的核心。当合作伙伴的集成失败时，平台通常先知道、也知道得更多——平台握着请求、状态码、投递尝试和时间戳——而合作伙伴只握着一个沉默的队列和一个愤怒的客户。下面的每个设计决策，要么在缩小这个差距，要么就是在让合作伙伴从外部反向工程你的平台。

Clarifying questions

澄清问题

Ask these before designing anything. Each answer moves a later ranking. Batch them - this is a tactical design task, not a strategy interview.

What can an integrator see today without asking you? Send the URL if any surface exists. "Nothing" is a valid and common answer.
Who breaks: engineers at a single customer integrating for their own use, commercial partners whose app runs across a fleet of installs, no-code and agent builders, or a mix? (see next section)
Classify the last 20 integration support tickets: request not found, webhook never arrived, silent data mismatch, auth or credential failure, rate-limit confusion. The shape of that pile decides which surface to build first.
Is every inbound request already tagged with an integration identity - API key, app or client ID - at ingestion? Aggregation is only ever as granular as the tag it is keyed on.
Do webhooks exist, and does the delivery machinery already record per-attempt outcomes? (decides the default rung in step 2)
When an integration breaks, does the partner lose money within hours, or is it noticed at the next reconciliation? (decides the notification posture in step 5)
What can you legally show? Do request and response bodies carry PII, secrets, or data belonging to another tenant, and who signs off on exposing them to a third party?
Deadline, payoff shape and effort ceiling: by what date must this be live, is it a one-off fix after a bad support quarter or a compounding platform surface you will run for years, and what can you spend - engineering weeks, log storage, an on-call rotation to answer the alerts you start sending?

Question 8 exists because every menu below diverges sharply on effort and on how long the payoff lasts. A hard deadline promotes whatever you already record and merely have to expose, while a compounding-surface mandate promotes the rungs that need new storage, new state, or a new outbound channel.

Question 7 can delete rungs outright rather than reorder them - a legal answer of "those bodies can never leave our tenancy boundary" removes payload-level logging from the menu instead of demoting it.

If your harness has persistent memory, store the surface's core decisions - which logs are exposed, the retention window, the identity the aggregation is keyed on, the alert thresholds and cooldown, the redaction rules - so a later run adding an event type or tuning an alert starts from the design instead of re-deriving it.

在设计任何东西之前先问这些。每个答案都会影响后面的排序。把它们一次性问完——这是一个战术性设计任务，不是策略面试。

集成方今天不问你就能看到什么？如果已存在任何界面，把URL发过来。“什么也没有”是一个常见且有效的答案。
谁会踩坑：单个客户的工程师为了自己使用而集成；商业合作伙伴的应用跑在一大批安装实例上；无代码构建者和Agent构建者；还是混合情况？（见下一节）
对最近20张集成支持工单做分类：找不到请求、webhook从未到达、静默数据不匹配、认证或凭据失败、限流困惑。这堆工单的形态决定先构建哪个界面。
每个入站请求在接收时是否已经被打上集成身份标签——API key、应用或客户端ID？聚合的粒度上限永远取决于它按什么键来打标签。
是否存在webhook，投递机制是否已经记录每次尝试的结果？（决定第2步的默认档位）
集成故障后，合作伙伴是在几小时内损失金钱，还是等到下一次对账才发现？（决定第5步的通知姿态）
从法律上你能展示什么？请求和响应体是否携带PII、机密，或其他租户的数据？谁负责批准把数据暴露给第三方？
截止日期、回报形态和投入上限：必须在什么日期前上线？是一次性的糟糕支持季度后的补救，还是要长期运营、持续复利的平台界面？你能花多少——工程周数、日志存储、一个用来应答你开始发送的告警的on-call轮值？

问题8之所以存在，是因为下面的每个菜单在投入成本和回报持续时间上差别极大。硬性截止日期会抬升那些你已经在记录、只需要暴露出来的选项；而复利型平台界面的要求则会抬升那些需要新存储、新状态或新出站通道的档位。

问题7可以直接删除某些档位，而不只是重排——法律上“这些请求/响应体永远不能离开我们的租户边界”会把负载级日志从菜单里整个拿掉，而不是降级。

如果你的harness有持久记忆，把该界面的核心决策存下来——暴露了哪些日志、保留窗口、聚合所依据的身份、告警阈值和冷却时间、脱敏规则——这样后续某次运行要增加事件类型或调整告警时，可以从设计出发，而不是重新推导一遍。

Integration ownership

集成所有权

How many end users sit behind the failing integration decides whether a dashboard is enough:

Single-account integrators - a customer's own engineers wiring your API into their systems. One account, one dashboard, one set of credentials. A good self-serve log surface genuinely finishes the job for them: they will go look when something breaks, because their own users are already complaining.
Partners and ISVs with a fleet of installs - their app runs across tens or thousands of customer accounts, and each install can fail independently. A per-account dashboard is structurally useless to them: nobody watches a thousand dashboards, so they need cross-install aggregation and push notification instead, or they learn about breakage from their own customers. This is the segment that makes steps 5 and 6 load-bearing rather than optional.
No-code and agent builders - assemble a flow, cannot read a stack trace, and will not correlate anything. They need one plain-language failure list scoped to their flow, and a "what do I do about it" line next to each entry. Detail that helps the other two segments reads as noise here.

Serve every segment your answer to question 2 names. A surface designed only for the first segment is the documented default failure: the platform ships logs, calls integration visibility done, and its partner base still finds out about outages from end users.

有多少终端用户坐在失败的集成后面，决定了“一个仪表盘”是否足够：

单账户集成方——客户自己的工程师把你的API接到他们的系统里。一个账户、一个仪表盘、一套凭据。一个好的自助日志界面确实能帮他们真正解决问题：东西坏了的时候他们会自己来看，因为他们的用户已经在抱怨了。
带一批安装实例的合作伙伴和ISV——他们的应用跑在几十或数千个客户账户上，每个安装都可能独立失败。按账户的仪表盘对他们来说结构性地没用：没有人会盯着一千个仪表盘看，所以他们需要跨安装的聚合和推送通知，否则就只能从他们自己的客户那里得知故障。正是这个群体，让步骤5和步骤6从可选项变成了承重墙。
无代码构建者和Agent构建者——他们会拼装一条流程，但读不了堆栈跟踪，也不会做任何关联排查。他们需要一个限定到自身流程、用白话写的失败列表，以及每条旁边的“我该怎么办”说明。对另外两个群体有用的细节，在这里只会被视为噪音。

问题2点到了哪类群体，你就要为哪类群体服务。只按第一类群体设计的界面，就是文档里写明的默认失败模式：平台上线了几个日志、宣称集成可见性做完了，而它的合作伙伴依然是从终端用户那里得知故障。

Workflow

工作流程

Audit what an integrator can see today.
Expose per-integration failure detail.
Make the correlation ID reach support.
Aggregate error rates per integration identity.
Decide the notification posture.
Build the provider-side fleet view.

Each step has a section below, in order.

审计集成方今天能看到什么。
暴露按集成的失败详情。
让关联ID能到达支持团队。
按集成身份聚合错误率。
决定通知姿态。
构建提供方侧的全局视图（fleet view）。

每一步在下面都有一节，按顺序排列。

1. Audit what an integrator can see today

1. 审计集成方今天能看到什么

Start from the support pile (question 3), not from an architecture diagram. For each of the last 20 integration tickets, answer one question: could the integrator have answered this themselves with a surface you could reasonably ship? The share that says yes is your baseline, and it is the number every later step is trying to move.

Walk each failure class separately: synchronous API errors, webhooks that never arrived, auth or credential expiry, rate-limit rejections, and silent data mismatches. Platforms routinely ship a surface for one and assume it covers the others.
Check the failure that produces no error at all: an integration that stopped calling you entirely. Nothing in an error log shows an absence, so a volume-drop signal is a separate design item, not a side effect of error logging.
Record retention honestly. A log an integrator cannot reach back to the moment of the incident is a log that generates a support ticket anyway.
Note who can see it. A surface that only the account owner can open fails the partner whose engineer is not a user of that account - an access-model gap, not a logging gap.

Write the audit as a table of failure class against current visibility, and carry it into every step below. The gaps are the build order.

从支持工单堆出发（问题3），而不是从架构图出发。对最近20张集成工单，回答一个问题：如果有一个你合理能上线的界面，集成方能否自己回答这个问题？回答“能”的比例就是你的基线，也是后续所有步骤试图移动的那个数字。

逐个失败类别走一遍：同步API错误、从未到达的webhook、认证或凭据过期、限流拒绝、静默数据不匹配。平台经常只为一个类别做了界面，然后假设它能覆盖其他所有类别。
检查那种根本不产生错误的失败：一个集成完全停止调用你了。错误日志里没有任何东西能表示“空无”，所以“调用量骤降”是一个独立的设计项，而不是错误日志的副产品。
诚实记录保留期。集成方够不到事发时刻的日志，就是一张无论如何都会变成支持工单的日志。
注意谁能看到它。只有账户所有者能打开的界面，会让那个实际维护集成的、却不是该账户用户的合作伙伴工程师看不到——这是访问模型的问题，不是日志的问题。

把审计写成一个失败类别对当前可见性的表格，并带着它进入下面每一步。那些空缺就是构建顺序。

2. Expose per-integration failure detail

2. 暴露按集成的失败详情

Three distinct surfaces exist here, and conflating them is the common design error. Stripe's developer dashboard is the reference implementation to copy the shape of, and it keeps them separate (sourced: Stripe docs):

Request log - every API call with its outcome. Answers "did my call work".
Event log - the business-level events those calls produced. Answers "what did it change".
Webhook delivery log - every delivery attempt per endpoint with the failure reason. Answers "did you tell me about it".

An integrator debugging in the dark needs whichever one matches their failure class.

Rungs, from what you can expose cheapest to what costs most to build:

value (support tickets deflected):

request log with per-request detail > delivery-attempt log > replay > failure counts only

effort:

replay > request log with per-request detail > delivery-attempt log > failure counts only

compliance cost:
```
request log with per-request detail > delivery-attempt log > replay == failure counts only
```
- exposing request and response bodies to a third party triggers a data-classification and redaction review, and it is close to irreversible: once integrators build tooling against a field set you cannot quietly withdraw it.

efficiency:

delivery-attempt log > request log with per-request detail > failure counts only > replay

Default rung: expose the delivery-attempt log you already write, when the platform sends webhooks. The delivery machinery records every attempt and its failure reason regardless. Exposing it is mostly an access-control and interface job, and it retires the single most support-expensive question in webhook platforms ("your webhook never arrived").

Where there are no webhooks, the request log is the default instead.

Move up to the full request log - per call: timestamp, endpoint, status code, error code, request ID, and as much of the payload as question 7 permits - as soon as the audit shows synchronous API failures in the ticket pile. This is the rung that covers every integrator segment rather than only the webhook consumers.

Failure counts alone are the floor rung: a number per integration with no detail behind it. It is cheap, and worth shipping the same week if nothing exists today. But understand what it buys: it tells a partner something is wrong and gives them no way to act, which converts a silent failure into a support ticket rather than into a fix.

The starved option: replay. Letting an integrator re-send a failed event or re-run a request recovers value no other rung measures - the deflection axis above does not count it. But it carries the highest effort on the menu: idempotency guarantees, a bounded window, and a clear relationship to your own retry schedule. It loses every efficiency round as a result.

Promote it when the audit shows integrators regularly losing events during their own downtime, which is exactly when a dashboard that only explains the loss is cold comfort.

Design it as independent of automatic retry. On Stripe, a failed event can be manually resent for up to 15 days after creation, and doing so does not cancel the platform's own retry schedule (sourced: Stripe docs). So a partner's manual replay never fights the platform's machinery.

This ranking is a default, not a law. Re-rank it against what you already know:

A platform with no webhooks has no delivery log to expose, and starts at the request log.
A team already storing full request/response bodies for its own debugging gets the request-log rung for the cost of an access-scoped view.
A "no bodies leave our boundary" answer to question 7 deletes payload detail from the menu, leaving metadata-only logging as the ceiling rather than a compromise.

See references/visibility-surfaces.md for the three-surface taxonomy expanded, the fields a log row has to carry, a good/bad log-row pair, replay semantics, and the tenant-isolation and redaction rules.

这里存在三个不同的界面，把它们混为一谈是常见的设计错误。Stripe的开发者仪表盘是值得照抄形状的参考实现，而它确实把三者分开了（来源：Stripe文档）：

请求日志（Request log）——每次API调用及其结果。回答“我的调用成功了吗”。
事件日志（Event log）——这些调用产生的业务级事件。回答“它改变了什么”。
Webhook投递日志（Webhook delivery log）——每个端点的每次投递尝试及失败原因。回答“你有没有告诉我”。

在黑暗中调试的集成方，需要的是与他们的失败类别匹配的那一个。

档位，从你能以最低成本暴露的，到建造起来最贵的：

价值（挡掉的工单数）：

带每次请求详情的请求日志 > 投递尝试日志 > 重放 > 仅失败计数

成本：

重放 > 带每次请求详情的请求日志 > 投递尝试日志 > 仅失败计数

合规成本：
```
带每次请求详情的请求日志 > 投递尝试日志 > 重放 == 仅失败计数
```
——把请求和响应体暴露给第三方会触发数据分级和脱敏审查，而且几乎不可逆：一旦集成方围绕某套字段建立了工具，你就不能悄无声息地把字段撤走。

效率：

投递尝试日志 > 带每次请求详情的请求日志 > 仅失败计数 > 重放

默认档位：暴露你本来就在写的投递尝试日志，当平台发送webhook时。投递机制无论如何都会记录每次尝试及其失败原因。把日志暴露出去主要是一个访问控制和接口工作，而且它一次性解决了webhook平台上最消耗支持工单的问题（“你的webhook从没到过”）。

如果没有webhook，默认档位就换成请求日志。

升级到完整请求日志——每次调用：时间戳、端点、状态码、错误码、请求ID，以及问题7允许范围内的尽可能多的负载——只要审计显示工单堆里有同步API失败。这个档位能覆盖所有集成方群体，而不只是webhook消费者。

只有失败计数是最低档：每个集成一个数字，后面没有任何细节。它便宜，如果今天什么都还没有，同一周内就值得上线。但要明白它买来的是什么：它告诉合作伙伴出事了，却不给他们任何行动手段，这会把一个静默失败变成一张支持工单，而不是一个修复。

**被饿死的选项：重放。**让集成方重新发送一个失败事件或重跑一次请求，能收回其他任何档位都衡量不到的价值——上面那条“挡掉工单”的坐标轴并没有把它算进去。但它是整个菜单里成本最高的活：幂等保证、有界时间窗口，以及和你自己重试调度之间明确的关系。因此它在每一轮效率比较中都会输。

当审计显示集成方经常在自己停机期间丢事件时，把它提上来——恰恰在那种时候，一个只是解释“你丢了”的仪表盘，一点安慰作用都没有。

把它设计成与自动重试相互独立。在Stripe上，失败事件可以在创建后最多15天内手动重发，而且这样做不会取消平台自己的重试调度（来源：Stripe文档）。所以合作伙伴的手动重放永远不会和平台的机制打架。

这个排名是默认值，不是铁律。拿你已经知道的情况重新排：

没有webhook的平台就没有投递日志可暴露，起点就是请求日志。
本来就把完整请求/响应体存下来给自己调试的团队，用一个按访问范围限定的视图就能拿到请求日志档位，成本很低。
对问题7回答“任何body都不能离开我们边界”，会把负载细节从菜单里删掉，于是只剩元数据日志成为天花板，而不是一种妥协。

关于三种界面的展开分类、日志行必须携带的字段、一组好/不好的日志行对比、重放语义，以及租户隔离和脱敏规则，见 references/visibility-surfaces.md。

3. Make the correlation ID reach support

3. 让关联ID能到达支持团队

Two identifiers, kept separate in your vocabulary:

Request ID - identifies one HTTP call and its response.
Correlation ID - identifies a whole multi-call transaction that may span many request IDs.

Most platforms need both and ship only the first.

The convention is

X-Request-Id

carrying an opaque random value - a UUIDv4 by default, since it is collision-safe under load and leaks nothing, unlike a sequential or timestamp-derived identifier. Named platform variants exist (GitHub's

request-id

, Vercel's

X-Vercel-Id

, Cloudflare's

Cf-Ray

) and are citations, not requirements. The W3C

traceparent

header is the standards-track alternative that carries a span hierarchy and sampling flag instead of a bare string.

value (a support ticket resolved in one lookup):

searchable in the developer-facing log > echoed in the response header > present in the error body > propagated to sub-processors

effort:
```
propagated to sub-processors > searchable in the developer-facing log > echoed in the response header == present in the error body
```
- that tie is genuine rather than an evasion: both write an identifier you already hold into the response you are already serialising, in the same code path, with no new storage behind either.

efficiency:

echoed in the response header > present in the error body > searchable in the developer-facing log > propagated to sub-processors

Default rung: echo the ID in the response headers of every response, success and failure alike. An integrator cannot paste an identifier into a support ticket if it was never returned to them, and this rung is usually a middleware change measured in hours. Success responses matter too - plenty of tickets are about a call that returned 200 and did the wrong thing.

Promote to searchable-in-the-log the moment step 2 ships. These two rungs are the pair that actually closes the loop: the header hands the integrator a string, the log lets them resolve it themselves before they ever write to you. Shipping the header alone produces a distinctive failure - integrators dutifully quoting an ID nobody outside your platform can look up.

The error body rung exists but belongs to a sibling:

samber/developer-platform-skills@api-error-design

owns the shape of the error payload, including whether the ID rides in it. Coordinate rather than redesign it here.

The starved option: propagation to sub-processors. Forwarding the same identifier onward when your platform calls a payment processor, carrier or other downstream provider, and keeping it on retries, is high effort and invisible until it matters. Promote it when your support answer today is "that failure came from our provider, we cannot see it" - at that point one lookup spanning the whole chain is the difference between a resolved ticket and a handoff.

This ranking is a default, not a law. Re-rank it against your stack:

A platform already emitting W3C trace context internally gets propagation nearly free, which flips the effort line.
A platform whose failures are overwhelmingly its own code gains little from propagation regardless of cost.

See references/correlation-id-conventions.md for header conventions, request-versus-correlation-ID modelling, propagation rules, the support-lookup workflow, and a negative example.

两个标识符，在你的词汇表里分开：

请求ID（Request ID）——标识一次HTTP调用及其响应。
关联ID（Correlation ID）——标识一个可能跨越多个请求ID的、多调用的完整事务。

大多数平台两者都需要，却只交付了第一个。

惯例是

X-Request-Id

，携带一个不透明的随机值——默认是UUIDv4，因为它在负载下无碰撞风险，且不像递增或时间戳派生的标识符那样会泄露信息。存在一些已命名的平台变体（GitHub的

request-id

、Vercel的

X-Vercel-Id

、Cloudflare的

Cf-Ray

），它们是被引用的例子，不是要求。W3C的

traceparent

头是标准轨道的替代方案，携带的是span层级和采样标记，而不是一个裸字符串。

价值（一张支持工单用一次查询就解决）：

可在面向开发者的日志中搜索 > 在响应头中回显 > 出现在错误体中 > 传播给子处理器

成本：
```
传播给子处理器 > 可在面向开发者的日志中搜索 > 在响应头中回显 == 出现在错误体中
```
——这个并列是真实的，不是回避：两者都是在你已经序列化的响应里、同一代码路径中写入一个你已经持有的标识符，后端都不需要新存储。

效率：

在响应头中回显 > 出现在错误体中 > 可在面向开发者的日志中搜索 > 传播给子处理器

默认档位：在每一个响应的响应头中回显这个ID，成功失败都一样。如果一个标识符从未返回给集成方，他们就没法把它粘到支持工单里，而这个档位通常只是一个以小时计量的中间件改动。成功响应同样重要——大量工单是关于一次返回200却做错了事的调用。

**步骤2一上线，立刻把“可在日志中搜索”提上来。**这两个档位才是真正闭合回路的一对：响应头递给集成方一个字符串，日志让他们在给你写信之前自己就能解析。只上线响应头会产生一种特有的失败——集成方老老实实引用一个ID，但平台之外没人能查。

错误体档位存在，但它属于一个相邻技能：

samber/developer-platform-skills@api-error-design

负责错误负载的形态，包括ID是否放在里面。这里要与它协调，而不是重新设计。

**被饿死的选项：向子处理器传播。**当你的平台调用支付处理器、承运商或其他下游提供方时，把同一个标识符继续传递下去，并在重试时保持，这是高成本的，而且在出事之前完全不可见。当你的支持团队今天的答案是“那个失败来自我们的提供方，我们看不到”时，把它提上来——那时，一次能跨整条链路的查询，就是“工单被解决”和“工单被转手”的区别。

这个排名是默认值，不是铁律。拿你的技术栈重新排：

一个内部已经在输出W3C trace context的平台，几乎免费就能获得传播，这会让成本线翻转。
一个失败绝大多数出自自身代码的平台，无论成本高低，从传播中得到的收益都很小。

关于头部约定、请求ID与关联ID的建模、传播规则、支持团队查询流程和一个反面例子，见 references/correlation-id-conventions.md。

4. Aggregate error rates per integration identity

4. 按集成身份聚合错误率

A platform-wide error rate can look healthy while one specific integration is failing 100% of its calls. Per-identity aggregation is what makes that visible, and the architecture is consistent across API-monitoring practitioner sources (sourced: Zuplo, Last9, dotcom-monitor, Odown):

Tag at ingestion, not after the fact - API key, app or client ID, attached to the request as it arrives. Retrofitting the tag later means your history is unqueryable for exactly the period you want to investigate.
Split by status-code family rather than one blended rate - 4xx (the integrator's payloads or auth), 5xx (yours), and 429 isolated on its own because rate-limit rejection is neither party's bug and has a different remediation. Blending them produces a number nobody can act on: you cannot tell a partner "your error rate is 8%" without saying whose fault the 8% is.
Compute a rolling rate per identity, and compare each integration against its own recent baseline rather than a global threshold - integrations differ so widely in traffic shape that one platform-wide percentage is either noisy for the small ones or blind to the large ones. Moesif's API monitoring documents both halves of this: status-code distribution recorded per endpoint and per customer in real time, and a "Dynamic Alert" that builds a historical baseline from a few days of data before alerting on deviation from it (sourced: Moesif docs).
Correlate against your own deploy timeline before concluding anything. A spike that lines up with your last release is your regression, and telling a partner otherwise is a trust cost you pay once. This is a named, shipped practice rather than a hypothetical - Datadog's Deployment Tracking overlays deploy and version markers on error-rate graphs specifically to answer whether a new release introduced the errors (sourced: Datadog docs).
Read distribution, not just volume - errors clustered around one time or one action pattern point to a logic bug or a bulk operation. Errors spread evenly point at infrastructure instead. This changes both who you notify and what you say.

429 specifically hands off to

samber/developer-platform-skills@api-rate-limit-policy

: the surfacing is yours, the quota design and headers are that skill's.

See references/alerting-and-notification.md for the aggregation schema, the status-code bucket definitions, and burn-rate detection windows.

平台级的错误率可能看着很健康，但某一个集成却在100%失败。按身份的聚合正是要让这一点可见，而且架构在API监控实践来源中是一致的（来源：Zuplo、Last9、dotcom-monitor、Odown）：

在接收时打标签，而不是事后补 ——在请求到达时就把API key、应用或客户端ID挂上去。事后补标签，意味着你的历史数据恰恰在你最想调查的那段时期无法被查询。
按状态码族拆分，而不是合成一个混合率 ——4xx（集成方的载荷或认证）、5xx（你的）、429单独存放，因为限流拒绝既不是任何一方的bug，而且补救方式也不同。把它们混在一起会产生一个谁都无法行动的数字：你不能跟合作伙伴说“你的错误率是8%”，却不说这8%是谁的错。
对每个身份计算滚动速率，并把每个集成与它自己最近的历史基线比较，而不是与一个全局阈值比较——集成之间的流量形态差异太大，一个平台范围的百分比不是对小集成太吵，就是对大集成太瞎。Moesif的API监控把这两半都文档化了：按端点和按客户实时记录状态码分布，以及一个“动态告警”，先从几天数据里建立历史基线，再针对对基线的偏离发出告警（来源：Moesif文档）。
**在下任何结论之前，先与你自己的部署时间线关联。**一个与你上次发布对齐的尖峰，就是你的回归；如果这时候还跟合作伙伴说是对方的错，这个信任代价你只会付一次。这是一个已命名、已上线的实践，而不是假设——Datadog的Deployment Tracking正是把部署和版本标记叠加在错误率图上，用来回答“新发布是否引入了这些错误”（来源：Datadog文档）。
看分布，而不只是看数量 ——错误集中在某个时间或某个动作模式，指向逻辑bug或批量操作；错误均匀铺开，则指向基础设施问题。这既改变你通知谁，也改变你通知什么。

429尤其要移交给

samber/developer-platform-skills@api-rate-limit-policy

：界面展示是你的，配额设计和响应头是那个技能的。

关于聚合schema、状态码桶定义和燃烧速率检测窗口，见 references/alerting-and-notification.md。

5. Decide the notification posture

5. 决定通知姿态

Aggregation without notification only helps the integrators who think to go look - which excludes the entire fleet segment from the audience split. But push notification is where this design most easily destroys itself: an alert stream that fires on every failure gets muted, and a muted channel delivers nothing at all.

value (partner learns before their own customers do):

threshold alert with cooldown > digest > immediate alert on state changes > pull-only dashboard

effort:

threshold alert with cooldown > digest > immediate alert on state changes > pull-only dashboard

efficiency:

digest > threshold alert with cooldown > immediate alert on state changes > pull-only dashboard

Default rung: a periodic digest per integration - a scheduled summary of failures by class, with links into the log from step 2. It needs an aggregation window and a scheduler, nothing more, and it is the rung that respects the partner's attention by default rather than by tuning.

The starved rung is threshold alerting with a cooldown - top of the value line, top of the effort line, and beaten on efficiency by the digest every round. Promote it when question 6 says the partner loses money within hours. The concrete mechanic to copy comes from Salesforce's Agentforce health monitoring (sourced: Salesforce): notify immediately on an error-rate spike or latency creep, then hold a cooldown - 30 minutes by default there - before the same integration can alert again.

The cooldown is what makes immediacy survivable. Without it, the same incident alerts continuously until someone mutes the channel permanently.

Immediate alerts on integration state changes - credentials revoked, integration disabled, subscription expired - are a separate, narrow rung that coexists with the others. These are single discrete events, not a stream, and they are the class where an hour of digest delay is genuinely too slow.

Per-failure immediate alerting on error volume is not on this menu. It is the alert-fatigue failure mode itself rather than a rung, and parking it at the bottom of a list is how it quietly reappears as scope later.

Ship the logging first, but do not mistake that for finishing. This is a documented real-world gap rather than a hypothetical one: Shopify's partner dashboard surfaces function error counts and billable-event errors while threshold-based email alerting on those errors remains a developer request rather than a shipped capability (sourced: Shopify dev docs and developer community). Logging existing without push alerting is the normal resting state of platforms that considered the job done at step 2.

This ranking is a default, not a law. Re-rank it against the user:

A platform that already runs an outbound notification channel to partners gets every push rung cheaper, which compresses the effort line.
A partner base of single-account integrators who watch their own systems closely can legitimately stop at the digest.
Question 8's effort ceiling decides how much threshold tuning you can commit to, because an alert you cannot afford to tune becomes an alert nobody reads.

Notification templates and the anti-fatigue rules are in references/alerting-and-notification.md.

没有通知的聚合，只对那些会想到“去看一眼”的集成方有用——这把整个“安装群”群体从受众里排除掉了。但推送通知也正是这个设计最容易自我摧毁的地方：每条失败都触发的告警流会被静音，而被静音的通道什么也投递不了。

价值（合作伙伴在它们自己的客户之前知道）：

带冷却的阈值告警 > 摘要 > 状态变化即时告警 > 仅拉取仪表盘

成本：

带冷却的阈值告警 > 摘要 > 状态变化即时告警 > 仅拉取仪表盘

效率：

摘要 > 带冷却的阈值告警 > 状态变化即时告警 > 仅拉取仪表盘

默认档位：按集成定期发送摘要 ——一份按类别分组的失败定时总结，附上能链到第2步日志的链接。它只需要一个聚合窗口和一个调度器，仅此而已；而且这个档位默认就是尊重合作伙伴注意力的，而不是靠调参。

被饿死的档位是带冷却的阈值告警 ——价值线顶端、成本线顶端，并且每一轮都在效率上输给摘要。当问题6说合作方在几小时内就损失金钱时，把它提上来。要照抄的具体机制来自Salesforce的Agentforce健康监控（来源：Salesforce）：在错误率尖峰或延迟爬升时立即通知，然后进入冷却期——在那里默认是30分钟——同一个集成才能再次告警。

冷却期正是让“即时”能存活的东西。没有它，同一次事故会持续告警，直到有人永久静音那个频道。

集成状态变化时的即时告警 ——凭据被吊销、集成被禁用、订阅过期——是一个独立的、更窄的档位，能与其他档位共存。这些是单个离散事件，不是一条流，而且是那种摘要延迟一小时都真的太慢的类别。

对错误_量_的每条失败即时告警，不在这个菜单上。它就是告警疲劳这个失败模式本身，而不是一个档位；把它放在列表底部，它就会悄悄以范围蔓延的形式重新出现。

先把日志上线，但别以为这就完了。这是一个被文档记录的真实世界缺口，不是假设：Shopify的合作伙伴仪表盘展示了函数错误计数和计费事件错误，然而针对这些错误的基于阈值的邮件告警，仍然是一个开发者呼声，而不是一个已上线能力（来源：Shopify开发文档和开发者社区）。日志存在、推送告警不存在，是那些认为在第2步就已经做完的平台常见的静止状态。

这个排名是默认值，不是铁律。拿具体用户重新排：

一个已经有面向合作伙伴的出站通知通道的平台，每个推送档位都会更便宜，这会压缩成本线。
一个由单账户集成方构成、会密切盯自己系统的合作伙伴群体，停在摘要档是合理的。
问题8的投入上限决定你能承诺多少阈值调优，因为你调不起的告警，会变成一个没人读的告警。

通知模板和反疲劳规则在 references/alerting-and-notification.md。

6. Build the provider-side fleet view

6. 构建提供方侧的全局视图

The same aggregation from step 4, read from your own side, answers a different question: which of our integrations are failing right now, and which partner should we contact? This is the surface your partner and support teams use, and it is what makes proactive outreach possible instead of reactive.

Rank integrations by blast radius, not by raw error count - a partner app failing across 200 installs matters more than a single account throwing more errors, and a flat leaderboard by volume inverts that.
Separate "broken since our last deploy" from "broken since their last release". You own the first outright. The second is the outreach case.
Include the silent-failure signal from step 1: an integration whose call volume dropped to zero shows up nowhere in an error ranking.
A weighted per-partner health score is a real shipped pattern rather than an invention. Third-party tooling built specifically for ISVs on Salesforce computes exactly this per managed package (sourced: ISVapp product documentation). Treat the specific weights as yours to set and to defend, not as something to borrow.

On borrowing SLO framing here, be careful. The published error-budget and SLO literature is written consumer-side, for a team measuring its own dependence on somebody else's API, and it does not transfer wholesale to a provider publishing reliability numbers for partners. The mirror-image framing - the platform being the trustworthy real-time source about integration health, so the partner does not have to build independent monitoring of you - is this skill's own construction, not a sourced practice.

Use it as a design intent. Do not present it to partners as an industry standard, and do not publish a per-partner reliability number until you can defend how it is computed.

See references/fleet-view-and-health-scoring.md for the fleet-view layout, health-score construction, and the SLO adaptation gap in full.

把第4步的同一个聚合，从你自己这边读出来，回答的是另一个问题：我们哪些集成现在正在失败，并且我们该联系哪个合作伙伴？这是你的合作伙伴团队和支持团队使用的界面，也是从被动响应变成主动外联的可能来源。

按爆炸半径给集成排序，而不是按原始错误数——一个在200个安装上失败的合作伙伴应用，比一个单个账户抛出更多错误更重要；一张按数量排的扁平排行榜会把这件事颠倒过来。
把“自从我们上次部署就坏了”和“自从他们上次发版就坏了”分开。前一种你完全自己负责。后一种才是外联场景。
把第1步里那个静默失败信号放进来：一个调用量掉到零的集成，在错误排行榜里哪儿都不会出现。
加权的按合作伙伴健康分是一个真实上线过的模式，不是发明。专门给Salesforce上的ISV构建的第三方工具，正是对每个托管包计算这个分数（来源：ISVapp产品文档）。把具体权重当成由你设定、并由你维护的东西，而不是借来的。

关于在这里借用SLO框架，要小心。已发表的错误预算和SLO文献是写给消费者侧的——一个团队衡量自己对别人API的依赖——它不能整体迁移到一个平台为合作伙伴发布可靠性数字的场景。镜像框架——也就是平台成为关于集成健康状态的、可信赖的实时来源，让合作伙伴不必为你建立独立的监控——是这一技能自己的构造，不是一个有来源的实践。

把它当设计意图来用。不要向合作伙伴把它说成行业标准，而且在你能够解释清楚计算方式之前，不要发布任何按合作伙伴的可靠性数字。

关于全局视图布局、健康分构造，以及SLO适配缺口的完整论述，见 references/fleet-view-and-health-scoring.md。

Failure modes

失败模式

Each of these is a direct audit finding:

Logging shipped, alerting called out of scope, and the platform still learns about partner outages from end users.
Alert fatigue: an alert per failure, no cooldown, and a partner base that has muted the channel - after which the platform is functionally back to pull-only while believing it notifies.
A request ID echoed in headers that no external surface can resolve, so integrators quote a string nobody but your engineers can look up.
A correlation ID dropped on downstream or retried calls, breaking the chain at exactly the hop the partner cannot see.
One blended error rate per integration, with 4xx, 5xx and 429 mixed together, so no notification can say whose problem it is.
A healthy global error rate hiding one integration failing every call.
Consumer-side SRE error-budget framing lifted wholesale into a partner-facing reliability promise the platform never designed to meet.
A dashboard scoped to the account owner, invisible to the partner engineer who actually maintains the integration.
Log retention shorter than the time it takes a partner to notice, so the evidence expires before the investigation starts.
Payloads exposed without a redaction pass, leaking PII, secrets, or another tenant's data into a third party's console.
Replay that fights the platform's own retry schedule, producing duplicate side effects the partner then has to reconcile.
An integration that stopped calling entirely and shows up in nothing, because absence produces no errors.

下面每一个都是一条直接的审计发现：

日志上线了、告警被宣布为超出范围，然后平台仍从终端用户那里得知合作伙伴故障。
告警疲劳：每条失败一个告警、没有冷却，合作伙伴群体已经静音了频道——之后平台实际等于回到仅拉取，却还相信自己有通知。
响应头里回显了一个请求ID，但没有任何外部界面能解析它，于是集成方引用一个只有你的工程师能查的字符串。
关联ID在下游调用或重试调用中被丢弃，链条恰好断在合作伙伴看不到的那一跳。
每个集成只有一个混合错误率，4xx、5xx和429混在一起，于是没有任何通知能说清是谁的问题。
一个健康的全局限率，掩盖了某一个集成每次调用都失败。
把消费者侧的SRE错误预算框架整体搬进一个面向合作伙伴的可靠性承诺，而平台从来就没有设计去满足它。
仪表盘限定到账户所有者，真正维护集成的合作伙伴工程师却看不到。
日志保留期比合作伙伴发现故障所需的时间还短，证据在调查开始前就过期了。
负载未经脱敏就暴露，把PII、机密或另一个租户的数据漏进第三方控制台。
重放与平台自己的重试调度打架，产生合作伙伴之后不得不对账的重复副作用。
一个集成完全停止调用，在什么都里都不出现，因为“缺席”不产生错误。

Measurement

衡量

Failure-class coverage - the share of failure classes from the step 1 audit that an integrator can now diagnose unaided. This is the gate: iterate until every class in your ticket pile has a surface, or a written reason it does not.
Request-ID resolvability - every error response carries an identifier that resolves in a surface the integrator can reach. Binary, and cheap to test by taking one ID from a real response and looking it up as an outside user would.
Self-serve deflection - re-run the step 1 classification on a fresh batch of tickets after shipping and compare. The direction is the signal. The level is not, so baseline from your own before-and-after rather than any published figure (self-set: no industry benchmark exists for this).
Time from failure to partner awareness - measured from first failure to the partner acting, not to the alert being sent. Track it per notification rung, since it is the number that justifies promoting a rung.
Alert action rate - the share of alerts a partner acts on. A falling rate is the alert-fatigue failure mode arriving. Treat a sustained decline as a signal to widen thresholds or lengthen the cooldown (self-set: threshold is yours to pick from your own baseline).

The first two are gates. The last three are trends to watch, never pass thresholds - none of them has a defensible industry number behind it.

失败类别覆盖率 ——第1步审计中的失败类别里，集成方现在能在无人协助下诊断的比例。这是一道门：持续迭代，直到你工单堆里的每个类别都有对应界面，或者有一条书面理由说明为什么没有。
请求ID可解析率 ——每个错误响应都携带一个标识符，并且能在集成方可触及的界面中解析出来。二值的，而且测试成本很低：从一个真实响应里取一个ID，像外部用户那样去查一次。
自助式工单分流效果 ——上线后拿一批新工单重新做一次第1步的分类并对比。方向就是信号；水平不是。所以要用你自己前后的基线做基准，而不是任何公开数字（自我设定：这个领域没有行业基准）。
从失败到合作伙伴感知的时间 ——从第一次失败算到合作伙伴采取行动，而不是算到告警发出。要按每个通知档位分别追踪，因为正是这个数字能证明某个档位值得提升。
告警行动率 ——合作伙伴对告警采取行动的比例。比例下降就是告警疲劳这个失败模式到场了。把持续下降当作信号，去放宽阈值或拉长冷却时间（自我设定：阈值由你自己的基线来定）。

前两个是门。后三个是趋势，要观察，千万不要设通过阈值——它们背后没有一个站得住脚的行业数字。

Invocation examples

调用示例

"Our partners find out their integration is broken when their customers complain - design what we should be showing them."
"Build the developer-facing request log for our API; we already store every request internally."
"We alert partners on every failed webhook and they've all muted us. Fix the notification design."
"Audit what an external integrator can actually see about their own failures on our platform today."
"Design a fleet view so our partner team knows which ISV apps are breaking across installs."

“我们的合作伙伴总是等客户来投诉才知道集成坏了——设计一下我们应该给他们看什么。”
“为我们的API构建面向开发者的请求日志；我们内部已经存了每个请求。”
“我们对每个失败的webhook都给合作伙伴发告警，他们把我们全静音了。把通知设计修好。”
“审计一下，今天一个外部集成方在我们的平台上，关于自己的失败到底能看到什么。”
“设计一个全局视图，让我们的合作伙伴团队知道哪些ISV应用正在跨安装地出问题。”

References

参考

references/visibility-surfaces.md - the request/event/delivery three-surface taxonomy, required log-row fields, a good/bad log-row pair, replay semantics, retention and access scoping, redaction rules.
references/correlation-id-conventions.md - request versus correlation ID, header conventions and platform variants, echo and propagation rules, the support-lookup workflow, a negative example.
references/alerting-and-notification.md - per-identity aggregation schema, status-code buckets, threshold and cooldown mechanics, multi-window burn-rate detection, notification templates, anti-fatigue rules.
references/fleet-view-and-health-scoring.md - provider-side fleet-view layout, blast-radius ranking, per-partner health-score construction, the consumer-side SLO adaptation gap.