Loading...
Loading...
Found 21 Skills
Help the user systematically identify and categorize failure modes in an LLM pipeline by reading traces. Use when starting a new eval project, after significant pipeline changes (new features, model switches, prompt rewrites), when production metrics drop, or after incidents.
You are an expert error analysis specialist with deep expertise in debugging distributed systems, analyzing production incidents, and implementing comprehensive observability solutions.
You are an expert error analysis specialist with deep expertise in debugging distributed systems, analyzing production incidents, and implementing comprehensive observability solutions.
Audit an LLM eval pipeline and surface problems: missing error analysis, unvalidated judges, vanity metrics, etc. Use when inheriting an eval system, when unsure whether evals are trustworthy, or as a starting point when no eval infrastructure exists. Do NOT use when the goal is to build a new evaluator from scratch (use error-analysis, write-judge-prompt, or validate-evaluator instead).
Advanced error analysis and pattern detection specialist for identifying, analyzing, and preventing software errors
Use production telemetry as INPUT to design new tests. Covers OpenTelemetry integration with tests, trace-based assertions, log-informed test creation, production-error analysis for coverage gaps, and telemetry-driven test prioritization. Use when: "trace-based testing," "design tests from logs," "OpenTelemetry assertions," "production errors point to test gaps," "telemetry-driven testing." Not for: safe rollout techniques (flags, canary) during release — use testing-in-production. Not for: scheduled post-deploy probes — use synthetic-monitoring. Not for: triaging CI failures — use ai-bug-triage. Related: testing-in-production, synthetic-monitoring, qa-metrics, ai-bug-triage.
Sentry JavaScript frontend bug pattern review based on real production errors. Use when reviewing React/TypeScript frontend code for common bug patterns. Trigger keywords: "javascript bug review", "frontend errors", "react error patterns", "sentry frontend bugs".
Debug experiment code with structured error analysis. Categorize errors, apply targeted fixes with retry logic, and use reflection to prevent recurring issues. Use when experiment code fails or produces incorrect results.
Run error analysis on a dataset. Build a review UI, select diverse samples, monitor annotations, and organize failure modes.
Build a structured taxonomy of failure modes from open-coded trace annotations. Use this skill whenever the user has freeform annotations from reviewing LLM traces and wants to cluster them into a coherent, non-overlapping set of binary failure categories (axial coding). Also use when the user mentions "failure modes", "error taxonomy", "axial coding", "cluster annotations", "categorize errors", "failure analysis", or wants to go from raw observation notes to structured evaluation criteria. This skill covers the full pipeline: grouping open codes, defining failure modes, re-labeling traces, and quantifying error rates.
Identifies bugs, analyzes errors, performs root cause analysis, and proposes fixes
Use when given a Sentry issue URL and you need to fetch exception details, stacktrace, and request context using sentry-cli (and Sentry API fallback when needed).