Auditing experiments and feature flags

read_data

list_data

Usage modes

Quick check (single entity)

1. Fetch the entity via

   read_data

   (e.g.,

   read_data("experiments", id)

   or

   read_data("feature_flags", id)

   ).
2. Apply the relevant checks from experiment checks or flag checks.
3. Report findings inline as markdown, grouped by severity (CRITICAL first, then WARNING, then INFO).
4. Include entity links as

   [Experiment: name](/experiments/id)

   or

   [Flag: key](/feature_flags/id)

   .

Scoped audit (one domain)

1. Bulk-fetch via

   list_data

   (e.g.,

   list_data("experiments")

   or

   list_data("feature_flags")

   ).
2. Run all checks for that domain against each entity.
3. Group findings by severity, then by entity.
4. Report as inline markdown.

Full audit (comprehensive)

1. Fetch all experiments via

   list_data("experiments")

   and all flags via

   list_data("feature_flags")

   .
2. Run all experiment checks and all flag checks.
3. Apply recurring patterns to identify patterns across multiple findings.
4. If there are more than 5 entities with findings, output as a notebook artifact via

   create_notebook

   for easier navigation. Otherwise report inline.

Output format

• Severity badge:

  🔴 CRITICAL

  ,

  🟡 WARNING

  , or

  🔵 INFO

• Check name: Which check produced this finding
• Entity link: Markdown link to the entity
• What's wrong: One-sentence description
• Action: What to do about it (see remediation actions)

🟡 WARNING — Flag integration · Experiment: checkout-redesign The linked feature flag is inactive (paused). Traffic is not being split. Action: Re-enable the flag or end the experiment.

Handling unavailable data

read_data

• Skip

  checkActivityHistory

  (experiment check) entirely.
• Skip the "toggle instability" and "never activated" sub-checks in flag lifecycle checks.
• In your report, note which checks were skipped and why:

  Skipped: Activity history checks (activity logs not available via current tools)

Partial failures

read_data

list_data

• Continue with the entities you could fetch.
• Report which entities could not be assessed and why.
• Do not silently omit entities from the audit.

Reference files

• Experiment checks — experiment configuration checks
• Flag checks — feature flag checks
• Finding types — severity and category definitions
• Recurring patterns — patterns across multiple findings
• Remediation actions — what to do about each finding