oodle-triage

Compare original and translation side by side

🇺🇸

Original

English
🇨🇳

Translation

Chinese

Oodle Triage — Right-Side Observability

Oodle Triage — 右侧可观测性

Turn an alert into evidence, not narrative. This skill gathers production context and files disciplined tickets. It closes the loop opened on the left: it reads the O11y Change Context manifest on recent PRs to connect a firing alert to the change that likely caused it.
Prime directive: separate what is proven from what is inferred at every step. A confirmed symptom, a leading hypothesis, and the exact missing evidence beats a tidy root-cause story built on correlation. Evidence before assertions.
将告警转化为证据,而非主观描述。该Skill会收集生产环境上下文信息并创建规范的工单。它闭合了左侧流程的循环:通过读取近期PR中的O11y Change Context清单,将触发的告警与可能导致其发生的变更关联起来。
核心准则: 在每一步都区分已证实推断的信息。一个已确认的症状、一个主要假设,以及明确缺失的证据,远胜于基于关联关系构建的看似完整的根因故事。先有证据,后有论断。

Modes

模式

  • Mode A — Triage context: input is a single alert, a tracker ticket id, or a symptom description → produce a context report and optionally update the ticket.
  • Mode B — Auto oncall triage: input is a time window (and a tracker parent issue) → enumerate everything that fired, dedupe, and file/update sub-issues. Designed to run on a
    /loop
    cadence.
If the input names one alert/ticket/symptom, use Mode A. If it asks to triage a window or "everything that fired," use Mode B (which reuses Mode A per alert).
  • 模式A — 告警上下文分类: 输入为单个告警、跟踪系统工单ID或症状描述 → 生成上下文报告,并可选择更新工单。
  • 模式B — 自动值班分类: 输入为时间窗口(以及跟踪系统的父级问题)→ 枚举所有触发的告警,进行去重,并创建/更新子工单。专为按
    /loop
    周期运行设计。
若输入指定单个告警/工单/症状,使用模式A。若要求对某个时间窗口或“所有触发的告警”进行分类处理,使用模式B(该模式会为每个告警复用模式A的逻辑)。

Discovery (do this first)

前置探索(请先执行此步骤)

Resolve backends by capability and tool shape, never by a hard-coded server name (names vary:
oodle-ai-us1
,
ap1
,
staging
,
dev
, or customer-custom).
  • Observability MCP — find tools for alerts/monitors, metric (PromQL) queries, logs, and traces. If the suite matches Oodle's shape, use the tight path: select the environment that owns the service, and load a matching
    load_skill__*
    workflow when one fits the problem (e.g. Kubernetes debugging, alert-noise analysis) before low-level calls.
  • Issue-tracker MCP — find the tracker in use. Linear is the worked example below; Jira is the analog (issue → sub-issue, states, comments map across).
  • If a backend is missing, do what the available signal allows and state the gap.
应根据能力和工具形态来选择后端服务,而非硬编码服务器名称(服务器名称各不相同:
oodle-ai-us1
ap1
staging
dev
或客户自定义名称)。
  • Observability MCP — 查找用于告警/监控、指标(PromQL)查询、日志和链路追踪的工具。若工具套件符合Oodle的形态,使用最优路径:在进行底层调用之前,选择服务所属的环境,并加载与问题匹配的
    load_skill__*
    工作流(例如Kubernetes调试、告警噪声分析)。
  • Issue-tracker MCP — 查找当前使用的跟踪系统。下文以Linear为例;Jira为同类替代方案(问题→子问题、状态、评论均可对应映射)。
  • 若缺少后端服务,根据可用信号执行可行操作,并说明缺失的内容。

Mode A — Triage context

模式A — 告警上下文分类

  1. Establish ground truth, in writing. If given a ticket, fetch it for scope. Pin service/component, cluster/env, pod(s), container, and the precise time window before querying anything. Convert the window to both UTC and the reporter's local zone — mixing them silently is a classic error.
  2. Confirm the symptom with hard signals before theorizing: restart/OOM reason, error rate, latency, resource usage vs limit, event logs. State the confirmed symptom in one line; everything after is explanation.
  3. Learn the instrumentation from the code. Read the relevant service code for the actual metric names, log strings, and span attributes — do not guess them. Note explicitly what is and isn't captured; knowing what is missing reveals which questions the telemetry cannot answer.
  4. Triangulate across telemetry — don't lean on one source. Metrics (grouped by every dimension that localizes blame: service, tenant, operation, pod, status), logs from caller and callee on the path (read ERROR/WARN and access logs, widen the filter if a reported log isn't found), traces end to end, and profiles for resource incidents.
  5. Correlate with recent changes (loop closes here). Find recent PRs touching the alerting service (via the GitHub or Linear MCP, or the
    gh
    CLI when no GitHub MCP is present) and read their O11y Change Context blocks.
    watch
    items and
    type
    focus the hypotheses (a
    watch: N+1 ...
    becomes the leading hypothesis);
    guarded-by
    confirms the alert↔code link;
    gaps
    explains evidence the telemetry was never built to provide.
  6. Track and label every finding
    Confirmed | Inferred | Unknown
    .
    A configured ceiling or default is not a measurement; co-occurrence is not causation. Actively try to falsify the leading hypothesis with one measurement — eliminating a candidate is worth more than adding support.
  7. When confirmation is impossible, stop and produce a ranked missing-evidence list: for each item, which hypothesis it would confirm/refute, where it likely lives, and whether it is accessible. Then ask to unblock the top item rather than guessing past it.
Output — a short, honest report: confirmed Symptom (with UTC+local timestamps), Attribution (who/what/where, to the extent proven), Leading hypothesis (proven vs inferred, plainly), Ruled out (with the measurement), Missing evidence (the ranked list). Offer to post it as a comment on the tracker ticket.
  1. 书面记录基准事实。 若给定工单,先获取工单以明确范围。在查询任何信息之前,确定服务/组件、集群/环境、Pod、容器以及精确的时间窗口。将时间窗口转换为UTC时区和上报者的本地时区——悄悄混用时区是典型错误。
  2. 先通过硬信号确认症状,再进行推测: 重启/OOM原因、错误率、延迟、资源使用量与限额对比、事件日志。用一句话陈述已确认的症状;后续内容均为解释说明。
  3. 从代码中了解埋点实现。 阅读相关服务代码,获取实际的指标名称、日志字符串和Span属性——切勿猜测。明确记录哪些信息已被捕获,哪些未被捕获;了解缺失的信息能明确遥测数据无法回答哪些问题。
  4. 跨遥测数据源交叉验证 — 不要依赖单一数据源。指标(按可定位问题的所有维度分组:服务、租户、操作、Pod、状态)、调用链中调用方和被调用方的日志(阅读ERROR/WARN和访问日志,若未找到上报的日志则放宽过滤条件)、全链路追踪,以及资源事件的性能剖析数据。
  5. 关联近期变更(至此闭合循环)。 找到涉及告警服务的近期PR(通过GitHub或Linear MCP,若无GitHub MCP则使用
    gh
    CLI),并阅读其中的O11y Change Context块。
    watch
    项和
    type
    用于聚焦假设(
    watch: N+1 ...
    会成为主要假设);
    guarded-by
    用于确认告警与代码的关联;
    gaps
    用于说明遥测数据原本就无法提供的证据。
  6. 为每个发现标记
    Confirmed | Inferred | Unknown
    (已确认 | 推断 | 未知)。
    配置的阈值或默认值并非实测数据;同时发生并不等同于因果关系。主动尝试通过一项实测数据推翻主要假设——排除一个候选假设比增加支持更有价值。
  7. 当无法确认时,停止操作并生成排序后的缺失证据列表: 针对每个条目,说明它能确认/推翻哪个假设、可能存在的位置,以及是否可访问。随后请求解锁优先级最高的条目,而非继续猜测。
输出 — 一份简短、真实的报告:已确认的症状(包含UTC+本地时间戳)、归因(已证实的责任人/事件/位置)、主要假设(明确区分已证实与推断)、已排除的假设(附带实测数据)、缺失证据(排序后的列表)。可将报告作为评论发布到跟踪系统的工单中。

Mode B — Auto oncall triage

模式B — 自动值班分类

Run Mode A's investigation per alert, wrapped in enumerate → suppress → dedupe → file. Match-check strictly precedes any create so reruns never double-file.
  1. Enumerate what fired in the window — the alert instances that reached the firing state. Never treat a pending/for-not-yet-satisfied condition as a fire: it sends no notification and must not be triaged or filed. Confirm a real fire from the alert-state signal or trigger history.
  2. Enrich each: still-active now, noise category (flapping/storm/perpetual/ auto-resolving/boundary) and trigger count over ~7d, muted? (an active muting rule whose matchers all match this alert's monitor + scope), and routed? (does the monitor page anyone, or is it unrouted).
  3. Suppress muted and unrouted alerts — they are not actionable oncall signal. Do not file them. If a stale ticket exists for a now-muted or now-unrouted alert, close it with a comment stating the reason (muting rule id + matchers, or unrouted monitor id). Suppress the specific instance, not a whole alert class — routing is per-monitor per-env.
  4. Collapse storms — many series of one monitor firing together become one logical alert → one ticket, not N.
  5. Investigate each distinct fired alert with Mode A, depth scaled to severity. Manifest correlation feeds probable-cause and suggested-fix.
  6. Dedupe & file against the tracker by semantic match anchored on the fingerprint (env + monitor + scope): no match → create; match open → comment the new occurrence; match closed → reopen and comment. Set priority from severity. Retry a failed write once; never leave a half-created duplicate. Rerunning the same window is idempotent.
对每个告警执行模式A的调查流程,并包裹在枚举→屏蔽→去重→创建工单的流程中。严格在创建工单前执行匹配检查,确保重复运行时不会重复创建工单。
  1. 枚举时间窗口内触发的告警 — 即进入**firing(触发)**状态的告警实例。切勿将pending(待触发)/未满足条件的状态视为触发: 此类状态不会发送通知,也无需进行分类处理或创建工单。通过告警状态信号或触发历史确认真实的触发事件。
  2. 为每个告警补充信息: 当前是否仍处于活跃状态、噪声类别(flapping/抖动、storm/风暴、perpetual/持续触发、auto-resolving/自动恢复、boundary/边界触发)、近7天的触发次数、是否已静音?(是否存在匹配该告警监控+范围的活跃静音规则)、是否已路由?(监控是否会通知任何人,还是未路由状态)。
  3. 屏蔽已静音和未路由的告警 — 这些不属于值班可处理的有效信号。请勿为其创建工单。若存在针对当前已静音或未路由告警的陈旧工单,关闭该工单并附上说明原因的评论(静音规则ID+匹配条件,或未路由的监控ID)。屏蔽的是特定告警实例,而非整个告警类别——路由配置是按每个监控、每个环境设置的。
  4. 合并风暴告警 — 同一监控连续触发的多次告警合并为一个逻辑告警 → 创建一个工单,而非N个。
  5. 使用模式A调查每个唯一的触发告警,调查深度根据告警级别调整。清单关联信息可提供可能的根因和建议修复方案。
  6. 基于fingerprint(指纹)进行语义匹配,在跟踪系统中去重并创建工单(指纹包含:env+monitor+scope):无匹配项→创建工单;匹配到未关闭工单→添加新触发事件的评论;匹配到已关闭工单→重新打开并添加评论。根据告警级别设置工单优先级。写入失败时重试一次;切勿留下半创建的重复工单。重复运行同一时间窗口的操作是幂等的。

Fingerprint block

Fingerprint块

Every filed ticket carries a stable fingerprint so future runs match it. Keep the format constant:
undefined
每个创建的工单都带有一个稳定的fingerprint,以便后续运行时能匹配到该工单。请保持以下格式不变:
undefined

Fingerprint

Fingerprint

  • env: <env>
  • monitor: <monitor id/name>
  • scope: <key=value scope labels>
  • links: <deep links to the backend>
undefined
  • env: <env>
  • monitor: <monitor id/name>
  • scope: <key=value scope labels>
  • links: <deep links to the backend>
undefined

Genericity vs Oodle-tight

通用性与Oodle专属实现

The discipline is vendor-neutral: confirmed-vs-inferred labeling, firing-only enumeration, suppressing muted/unrouted noise, storm collapse, fingerprint dedup, and manifest correlation apply to any backend. The mechanics are backend- specific: Oodle exposes an
ALERTS
-style state metric,
muting_rules
with typed matchers, and a monitor routing flag; a different vendor's MCP supplies its own equivalents. Use the equivalent where it exists; skip a step where the concept is genuinely absent — never invent a mechanic a backend does not have.
For the Oodle CLI path when deeper queries help, lean on oodle-metrics, oodle-logs, and oodle-traces rather than restating them here.
流程规范是厂商无关的:已确认/推断标记、仅枚举触发状态的告警、屏蔽已静音/未路由的噪声、合并风暴告警、fingerprint去重以及清单关联,这些规则适用于任何后端服务。实现机制则是后端专属的:Oodle提供
ALERTS
格式的状态指标、带类型匹配器的
muting_rules
以及监控路由标记;其他厂商的MCP会提供各自的等效功能。若存在等效功能则使用;若概念完全不存在则跳过该步骤——切勿编造后端不具备的机制。
若需要通过Oodle CLI进行更深层次的查询,请依赖oodle-metricsoodle-logsoodle-traces,而非在此重复其内容。