A/B Test Calculator

Plan first, then read the result

The two halves of this tool map to the two moments of an experiment. Before you start, the sample-size planner tells you how much traffic you need to detect the lift you care about — commit to that number. After the test reaches it, the significance checker tells you whether the difference is real or noise.

Why "no peeking" matters

If you stop a test the instant it crosses 95%, you will be wrong far more than 5% of the time — repeatedly checking a running test is multiple comparisons in disguise. Fix the sample size up front and only judge the result once.

From experiment to insight

This calculator works for any test. If you want the conversion events behind it — funnels, segments, and per-user timelines — Pug is open-source product analytics that captures the events your experiments measure.

Frequently asked questions

How many visitors do I need for an A/B test?

Enter your baseline conversion rate and the minimum lift you want to detect. The calculator returns the visitors needed per variant for a two-sided two-proportion test at your chosen confidence and power. Smaller effects and higher confidence require more traffic.

What is statistical significance?

Significance is the probability your observed difference is not just noise. A result is conventionally "significant" at 95% confidence when the p-value is below 0.05 — meaning under 5% chance the difference happened by luck.

What is statistical power?

Power is the chance of detecting a real effect when one exists. 80% is the common default: if there truly is a lift, an 80%-powered test finds it 80% of the time. Higher power needs more sample.

Should I stop a test as soon as it hits 95%?

No. "Peeking" and stopping at the first significant moment inflates false positives. Decide your sample size up front (use the planner above), run the full test, then read the result.