Oodle Triage — Right-Side Observability

Prime directive: separate what is proven from what is inferred at every step. A confirmed symptom, a leading hypothesis, and the exact missing evidence beats a tidy root-cause story built on correlation. Evidence before assertions.

Modes

• Mode A — Triage context: input is a single alert, a tracker ticket id, or a symptom description → produce a context report and optionally update the ticket.
• Mode B — Auto oncall triage: input is a time window (and a tracker parent issue) → enumerate everything that fired, dedupe, and file/update sub-issues. Designed to run on a

  /loop

  cadence.

Discovery (do this first)

oodle-ai-us1

ap1

staging

dev

• Observability MCP — find tools for alerts/monitors, metric (PromQL) queries, logs, and traces. If the suite matches Oodle's shape, use the tight path: select the environment that owns the service, and load a matching

  load_skill__*

  workflow when one fits the problem (e.g. Kubernetes debugging, alert-noise analysis) before low-level calls.
• Issue-tracker MCP — find the tracker in use. Linear is the worked example below; Jira is the analog (issue → sub-issue, states, comments map across).
• If a backend is missing, do what the available signal allows and state the gap.

Mode A — Triage context

1. Establish ground truth, in writing. If given a ticket, fetch it for scope. Pin service/component, cluster/env, pod(s), container, and the precise time window before querying anything. Convert the window to both UTC and the reporter's local zone — mixing them silently is a classic error.
2. Confirm the symptom with hard signals before theorizing: restart/OOM reason, error rate, latency, resource usage vs limit, event logs. State the confirmed symptom in one line; everything after is explanation.
3. Learn the instrumentation from the code. Read the relevant service code for the actual metric names, log strings, and span attributes — do not guess them. Note explicitly what is and isn't captured; knowing what is missing reveals which questions the telemetry cannot answer.
4. Triangulate across telemetry — don't lean on one source. Metrics (grouped by every dimension that localizes blame: service, tenant, operation, pod, status), logs from caller and callee on the path (read ERROR/WARN and access logs, widen the filter if a reported log isn't found), traces end to end, and profiles for resource incidents.
5. Correlate with recent changes (loop closes here). Find recent PRs touching the alerting service (via the GitHub or Linear MCP, or the

   gh

   CLI when no GitHub MCP is present) and read their O11y Change Context blocks.

   watch

   items and

   type

   focus the hypotheses (a

   watch: N+1 ...

   becomes the leading hypothesis);

   guarded-by

   confirms the alert↔code link;

   gaps

   explains evidence the telemetry was never built to provide.
6. Track and label every finding

   Confirmed | Inferred | Unknown

   . A configured ceiling or default is not a measurement; co-occurrence is not causation. Actively try to falsify the leading hypothesis with one measurement — eliminating a candidate is worth more than adding support.
7. When confirmation is impossible, stop and produce a ranked missing-evidence list: for each item, which hypothesis it would confirm/refute, where it likely lives, and whether it is accessible. Then ask to unblock the top item rather than guessing past it.

Mode B — Auto oncall triage

1. Enumerate what fired in the window — the alert instances that reached the firing state. Never treat a pending/for-not-yet-satisfied condition as a fire: it sends no notification and must not be triaged or filed. Confirm a real fire from the alert-state signal or trigger history.
2. Enrich each: still-active now, noise category (flapping/storm/perpetual/ auto-resolving/boundary) and trigger count over ~7d, muted? (an active muting rule whose matchers all match this alert's monitor + scope), and routed? (does the monitor page anyone, or is it unrouted).
3. Suppress muted and unrouted alerts — they are not actionable oncall signal. Do not file them. If a stale ticket exists for a now-muted or now-unrouted alert, close it with a comment stating the reason (muting rule id + matchers, or unrouted monitor id). Suppress the specific instance, not a whole alert class — routing is per-monitor per-env.
4. Collapse storms — many series of one monitor firing together become one logical alert → one ticket, not N.
5. Investigate each distinct fired alert with Mode A, depth scaled to severity. Manifest correlation feeds probable-cause and suggested-fix.
6. Dedupe & file against the tracker by semantic match anchored on the fingerprint (env + monitor + scope): no match → create; match open → comment the new occurrence; match closed → reopen and comment. Set priority from severity. Retry a failed write once; never leave a half-created duplicate. Rerunning the same window is idempotent.

Fingerprint block

## Fingerprint
- env: <env>
- monitor: <monitor id/name>
- scope: <key=value scope labels>
- links: <deep links to the backend>

Genericity vs Oodle-tight

ALERTS

muting_rules