Designing a Trustworthy A/B Test
Running an experiment is easy. Running one whose result you can trust is not. This is a hands-on toolkit for the parts that actually decide whether an A/B test is honest: powering it correctly before you start, testing significance properly when it ends, and avoiding the two mistakes that quietly ruin most experiments, peeking and ignoring variance reduction. Everything below computes live in your browser.
1. Design it
How many users do you need before you press go? Set the effect you care about and the calculator returns the sample size that gives you a fair shot at detecting it.
Smaller effects cost dramatically more traffic: halving the detectable effect roughly quadruples the sample you need. That tradeoff is the whole game.
2. Analyze it
The experiment is done. Did it actually work, or is it noise? Enter the counts and this runs a two-proportion z-test with a confidence interval, the honest verdict.
The two traps most experiments fall into
Peeking
Checking for significance repeatedly and stopping the moment you see p < 0.05 is the most common way to ship a fake win. Each peek is another chance to get unlucky. I simulated A/A tests (no real difference) and let them peek: the false-positive rate climbs from the intended 5% to over - after twenty looks. Fix it with sequential testing or a pre-committed sample size.
Ignoring variance reduction
CUPED uses a pre-experiment covariate (a user's prior behavior) to strip out noise you already could have predicted. In simulation it cut the variance of the effect estimate by -, worth roughly a - bigger sample for free. Same traffic, tighter answer, faster decisions.
How it works
- Power & sample size. Given a baseline rate, a minimum detectable effect, alpha, and power, the tool solves the two-proportion sample-size formula for the users needed per arm, then divides by your traffic to estimate a runtime.
- Significance. Results are judged with a two-proportion z-test: a pooled standard error for the p-value, an unpooled one for the confidence interval on the absolute lift. The interval, not just the p-value, is what tells you whether the effect is worth shipping.
- Peeking. Monte Carlo A/A simulations show how repeated looks inflate the false-positive rate far past the nominal 5%, and why a fixed horizon or a sequential test is needed.
- CUPED. A simulation with a correlated pre-period covariate demonstrates the variance reduction (about one minus the squared correlation), the cheapest way to make an experiment more sensitive.