Loading...
Loading...
A/B test design and experiment planning for paid advertising. Structured hypothesis framework, statistical significance calculator, test duration estimator, sample size calculator, and platform-specific experiment setup guides (Meta Experiments, Google Experiments, LinkedIn A/B). Use when user says A/B test, split test, experiment design, test hypothesis, statistical significance, sample size, or test duration.
npx skill4agent add agricidaniel/claude-ads ads-testIF we [change/action]
THEN [metric] will [increase/decrease] by [estimated %]
BECAUSE [reasoning based on data or insight]
Example:
IF we replace polished product shots with UGC creator videos
THEN Meta CTR will increase by 25-40%
BECAUSE Andromeda prioritizes diverse creative formats and UGC consistently outperforms polished in 2025-2026 benchmarksRequired Sample Size (per variant):
n = (Z_alpha + Z_beta)^2 × 2 × p × (1-p) / MDE^2
Where:
- Z_alpha = 1.96 (for 95% confidence)
- Z_beta = 0.84 (for 80% power)
- p = baseline conversion rate
- MDE = minimum detectable effect (relative %)
Simplified lookup:| Baseline CVR | 5% MDE | 10% MDE | 20% MDE | 30% MDE |
|---|---|---|---|---|
| 1% | 612,000 | 153,000 | 38,300 | 17,000 |
| 2% | 302,400 | 75,600 | 18,900 | 8,400 |
| 5% | 116,800 | 29,200 | 7,300 | 3,200 |
| 10% | 55,200 | 13,800 | 3,450 | 1,530 |
| 20% | 24,600 | 6,150 | 1,540 | 680 |
Duration = Required Sample Size / Daily Traffic per Variant
Minimum duration: 7 days (capture weekly patterns)
Maximum recommended: 28 days (avoid seasonal drift)
Learning phase: Google 7-14 days, Meta 3-7 days, LinkedIn 7-14 days
Inputs needed:
- Daily impressions or clicks
- Number of variants (2 = A/B, 3+ = multivariate)
- Baseline conversion rate
- Minimum detectable effect desired| Daily Clicks | 2% CVR, 20% MDE | 5% CVR, 20% MDE | 10% CVR, 20% MDE |
|---|---|---|---|
| 100 | 189 days | 73 days | 35 days |
| 500 | 38 days | 15 days | 7 days |
| 1,000 | 19 days | 7 days | 4 days* |
| 5,000 | 4 days* | 2 days* | 1 day* |
## A/B Test Plan
### Hypothesis
IF [change]
THEN [metric] will [direction] by [amount]
BECAUSE [reasoning]
### Test Design
| Parameter | Value |
|-----------|-------|
| Platform | [platform] |
| Test Type | [A/B / Multivariate] |
| Variable | [what's being changed] |
| Control | [current state] |
| Variant | [proposed change] |
| Primary Metric | [KPI] |
| Traffic Split | [50/50 / other] |
### Sample Size & Duration
| Metric | Value |
|--------|-------|
| Baseline CVR | [X%] |
| MDE | [X%] |
| Required Sample | [N per variant] |
| Daily Traffic | [N clicks/day] |
| Est. Duration | [X days] |
| Min Duration | 7 days |
### Success Criteria
- Winner declared at 95% confidence
- [Primary metric] improvement of [X%]+ sustained over [Y] days
- No negative impact on [secondary metric]
### Setup Instructions
[Platform-specific step-by-step]