This skill answers: My PostHog experiment results look wrong, biased, or empty — what's going on?

Match the user's complaint in the dispatch table, then read the matching reference file for the diagnostic.

Each diagnostic in the reference files is tagged , , or based on how strongly it's verified — is verified directly in PostHog code, is partially or team-source verified, describes SDK/external behavior that wasn't verified here. Treat items as hypotheses to test, not facts to assert.

If the user refers to an experiment by name or description, load the skill first to resolve it to a concrete ID.

Call and pull these fields. They are inputs for almost every diagnostic:

Before asking the user clarifying questions, pull the diagnostic snapshot in references/diagnostic-snapshot.md. Most diagnostics in this skill can be confirmed or ruled out from that data without an interview.

$multiple

If the symptom is unclear, ask one clarifying question before picking. Most diagnostics have different fixes — do not guess.

After matching the symptom in Step 2 and reading the relevant reference file(s), list each diagnostic that applies before recommending an action.

Surface co-occurring mechanisms independently — even when one is more salient, don't collapse them into a single "wait" or "fix" recommendation. Different mechanisms have different fixes: a systematic bias (e.g. uneven-split + Exclude) doesn't resolve by waiting; a statistical pattern (e.g. small-sample variance) does. Bundling them leaves the bias in place after the user follows the bundled advice.

Only list mechanisms that have a path to verification in the project state — config (from ), snapshot data, activity log, or repo source. Config-derived mechanisms count: an 80/20 split with default is visible in and is therefore enumerable. Naming a mechanism with no source (e.g. SRM when the snapshot shows a clean variant ratio) is not.

Variants don't look balanced, one variant looks biased, the in-app warning banner appeared, or users are showing up under multiple variants. Covers the uneven-split + Exclude interaction, SRM, identity fragmentation, bootstrap × mismatch, and flag/experiment state inconsistency.

→ See references/bias-and-skew.md

A frequent pain point. Covers SDK call (wrong evaluation method, timing, dedup), exposure capture (custom event missing variant property, required properties, ad-blockers), and exposure-criteria match (test-account filter, eligibility ordering, events firing before exposure).

→ See references/empty-experiment.md

Significance flipping, A/A test showing significance, Bayesian vs Frequentist confusion, multiple comparisons, low-volume variance, peeking / early stopping. Includes the legacy stats issue (A/A tests historically over-fired before the new Bayesian module) and how the win-probability methodology changed in Jan 2025 (single test vs control, not control vs all variants).

→ See references/interpretation.md

The experiment page applies an exposure scope, exclusion, test-account filter, and date range that ad-hoc SQL almost never replicates. Covers funnel attribution (only first→last step counts for stats), breakdowns (read from the exposure event, not the metric event), the "sum of revenue" mean-of-per-user confusion, and the recordings-panel-vs-stats divergence.

→ See references/numbers-vs-sql.md

Increasing rollout is safe; decreasing is caution; changing the variant split is an anti-pattern; adding metrics mid-run is p-hacking; ship-variant can rewrite the flag in surprising ways; reset clears results not the flag. Also covers retention-metric quirks (first-event-must-be-after-exposure design), "matured users" filtering, and long-term vs short-term metric divergence.

→ See references/mid-run-changes.md

Surface diagnostics first (Step 3). Then recommend — but scope what you recommend to what the experiment's current state permits.

On a stopped or archived experiment, don't preemptively offer reversal of a state mutation (ship-variant flag rewrite, manual flag edit, reset, archive). If the user asks "why did X happen?", explain X — don't append a "here's how to undo it" coda. That pattern assumes intent the user didn't signal. Conditional offers like "if this wasn't intended, you could…" or "want me to revert it?" count as preemptive too — only the user explicitly naming the reversal action ("how do I undo this?", "can I roll back ship-variant?", "how do I get the 50/50 split back?") is a request to surface reversal mechanics.

Use consistent terminology: variant split (between variants) is distinct from rollout (overall % entering); the exposure event is distinct from a custom exposure event; the Exclude / First seen options control multivariate handling, not exposure.