A/B Test Calculator

// Significance, in the sense that matters

A conversion rate that went from 10% to 13% has not necessarily improved. Split the same unchanged page into two random halves and their rates will still differ, because visitors arrive one at a time and conversion is noisy. The question a significance test answers is narrow and specific: if the two variants were truly identical, how often would random chance alone produce a gap at least this big? That number is the p-value.

A p-value of 0.035 means a gap this large would turn up about 3.5% of the time between two identical pages. At the conventional 95% bar you would call that significant, because 3.5% is below the 5% you agreed to tolerate. It does not mean there is a 96.5% chance the variant is better, and it does not tell you the lift is worth shipping — only that the gap is hard to explain as noise.

This calculator runs a two-tailed, pooled two-proportion z-test, the same test behind most mainstream A/B tools. Two-tailed matters: it tests whether the variants differ at all, in either direction, rather than quietly assuming your new version is the better one and halving the p-value in your favour.

// Read the interval, not just the verdict

The significant / not significant line is the least informative thing on the results panel. The confidence interval is the useful part, because it shows the range of true differences consistent with your data. An interval of +0.2% to +5.8% says the variant is probably better but the size of the win is still wide open. An interval of −0.1% to +6.1% crosses zero, which usually accompanies a not-significant verdict.

Usually, not always. The two are computed from different standard errors on purpose: the test pools both variants, because it asks what would happen if they were identical, while the interval does not pool, because it describes the gap you actually observed. Those agree comfortably away from the threshold and can disagree right at it — a borderline result can come back not significant while its interval sits just clear of zero. When they conflict, you are looking at a test that is too close to call, which is itself the answer.

Intervals narrow with traffic, not with time. If yours is uselessly wide, no amount of staring at the dashboard fixes it — only more visitors do. That is the honest reason a test that has been running for a week without a clear answer usually needs more sample, not a decision.

// Size the test before you run it, not after

The single most common way A/B tests go wrong is peeking: watching the dashboard and stopping the moment the numbers cross into significance. Every time you check, you give random noise another chance to wander across the line, and a test checked daily will hit 95% by luck far more often than 5% of the time. The fix is mechanical — decide the sample size up front, run to it, then look.

The planning mode does that arithmetic. Give it your current conversion rate and the smallest lift you would actually act on, and it returns the visitors per variant you need. The minimum detectable effect is a business decision, not a statistical one: detecting a 1% relative improvement takes roughly a hundred times the traffic of detecting a 10% one, and if you would not ship a 1% lift anyway, there is no reason to pay for the sample that could prove it.

Power is the other half. At 80% power — the usual default — a real effect of the size you specified is detected 80% of the time, which also means a genuine winner slips through undetected one run in five. Raising power to 90% cuts those misses at the cost of more traffic.

// Where this test stops being valid

The z-test uses a normal approximation to the binomial. It is accurate for the traffic volumes real tests run at, and it drifts when either outcome is very thin — fewer than about five conversions, or fewer than about five non-conversions, on either side. The tool says so in those cases rather than presenting a confident-looking number it cannot support. For the same reason it reports a p-value below 0.0001 as an inequality rather than rounding it to zero: no normal approximation earns the right to print certainty.

The maths also assumes each visitor is counted once, assigned at random, and measured over the same window on both sides. Tests that split by day, that let one person land in both buckets, or that run one variant through a weekend the other missed, break that assumption in ways no calculator can detect or repair. The arithmetic here is only ever as sound as the experiment that produced the counts.

// fair questions

What counts as statistically significant?

By convention, a p-value below 0.05 — the 95% confidence bar. That is a shared default, not a law of nature: 90% is a reasonable bar for a low-cost reversible change, and 99% is sensible when shipping the variant is expensive or hard to undo. Pick the level before you look at the data, not after.

Does a significant result mean the variant is better?

It means the gap is hard to explain as random chance. It does not mean the effect is large, durable, or worth shipping. Read the confidence interval to see how big the true difference plausibly is — a significant result whose interval runs from +0.1% to +0.4% is real and probably not worth the deploy.

Can I stop the test as soon as it hits 95%?

No, and this is the most common way A/B tests produce false winners. Checking repeatedly gives noise repeated chances to cross the line, so a peeked test reaches 95% by luck far more often than 5% of the time. Use the planning mode to set a sample size, run to it, then decide.

What is the difference between absolute and relative lift?

Absolute lift is the gap in percentage points: 10% to 13% is +3 points. Relative lift is the change as a proportion of where you started: the same move is +30%. Marketing decks quote the relative number because it sounds bigger; sample-size maths depends on the absolute one. This tool shows both.

Why does detecting a small lift need so much traffic?

Required sample scales with the inverse square of the effect size, so detecting an effect half as large takes about four times the visitors. That is why choosing a minimum detectable effect you would genuinely act on matters more than any other input on the page.

Do my numbers leave this page?

No. Every calculation runs in your browser. Nothing you type is uploaded, stored, or logged, and the page works with the network disconnected.