measure-experiment-design

Original：🇺🇸 English

Translated

Designs an A/B test or experiment with clear hypothesis, variants, success metrics, sample size, and duration. Use when planning experiments to validate product changes or test hypotheses.

5installs

Sourceproduct-on-purpose/pm-skills

Added on2026-04-29

NPX Install

npx skill4agent add product-on-purpose/pm-skills measure-experiment-design

SKILL.md Content

View Translation Comparison →

Experiment Design

An experiment design document defines all parameters needed to run a rigorous A/B test or controlled experiment. It ensures the team aligns on what you're testing, how you'll measure success, and how long to run the test before drawing conclusions. Good experiment design prevents common pitfalls: underpowered tests, unclear success criteria, and decisions based on noise rather than signal.

When to Use

Before launching an A/B test to validate a product change
When testing a hypothesis that requires quantitative validation
After solution design to validate assumptions before full rollout
When stakeholders want data-driven evidence for a decision
To establish a culture of experimentation and learning

Instructions

When asked to design an experiment, follow these steps:

Articulate the Hypothesis Write a clear, testable hypothesis in the format: "We believe [change] for [users] will [outcome] as measured by [metric]." One hypothesis per experiment . if you're testing multiple things, run multiple experiments.
Define the Variants Describe the control (current experience) and treatment (new experience) in sufficient detail. Include screenshots, mockups, or precise descriptions so anyone can understand what users will see.
Choose Primary and Secondary Metrics Select one primary metric that will determine success or failure. Add 2-3 secondary metrics to understand the broader impact. Include guardrail metrics to catch unintended negative effects.
Calculate Sample Size Determine how many users you need per variant to detect your minimum detectable effect (MDE) with statistical significance. Specify your significance level (typically 0.05) and power (typically 0.80).
Estimate Duration Based on sample size and available traffic, calculate how long the experiment needs to run. Account for weekly patterns . avoid ending mid-week if behavior varies by day.
Define Targeting and Allocation Specify which users are eligible for the experiment and how traffic is split between variants. Document any exclusions (e.g., employees, specific segments).
Set Success Criteria Define upfront what constitutes a win, a loss, or an inconclusive result. This prevents post-hoc rationalization and moving goalposts.
Document Risks and Mitigations Identify what could go wrong and how you'll detect/address it. Include monitoring plans and rollback criteria.

Output Format

Use the template in

references/TEMPLATE.md

to structure the output.

Quality Checklist

Before finalizing, verify:

Hypothesis is falsifiable and specific
Only one primary metric is defined
Sample size calculation is documented with assumptions
Duration accounts for traffic patterns and statistical requirements
Success criteria are defined before the experiment starts
Guardrail metrics protect against unintended harm

Examples

See

references/EXAMPLE.md

for a completed example.