A/B Test Significance Calculator — P-Value, Sample Size & Chance to Win

How A/B testing actually works

An A/B test splits visitors randomly between the current version (control, A) and a changed version (B), then counts how many in each group did the thing you care about — bought, signed up, clicked. Random assignment is the whole trick: because the two groups differ only by the change (plus luck), a persistent gap in conversion rates can be credited to the change itself. But luck does not disappear — it shrinks, slowly, as visitors accumulate. Even if A and B were truly identical, B would finish ahead in about half of all tests, sometimes by a margin that looks decisive. Statistical significance is the discipline of asking: if there were really no difference, how often would chance alone produce a gap at least this large? That probability is the p-value. At the standard 95% confidence level you call a result significant when the p-value falls below 0.05 — accepting that, run this way forever, about 1 in 20 "wins" will be a mirage. The calculator above runs the standard two-proportion z-test: it pools both groups to estimate the standard error under the null hypothesis, divides the observed difference by it to get the z-score, and converts that to a p-value. The confidence interval uses each group's own rate instead, because an interval should measure the difference as it is, not as the null imagines it.

A worked example, computed by this exact test

Variant A had 10,000 visitors and 500 conversions — 5.00%. Variant B had 10,000 visitors and 580 conversions — 5.80%. That is an absolute difference of +0.80 percentage points and a relative uplift of +16%. The pooled standard error is about 0.32 points, so z = 0.80 ÷ 0.32 ≈ 2.50, and the two-sided p-value is 0.0123 — below 0.05, so B wins at 95% confidence. The 95% confidence interval for the difference runs from about +0.17 to +1.43 points: B is very likely better, but the true lift could be a fifth of the one you measured. The Bayesian reading of the same data gives B a 99.4% chance of truly beating A, with an expected loss of about 0.001 points if you ship it and are wrong. Load the worked example in the calculator and every one of these numbers is reproduced — then change the sample to 1,000 visitors per variant with the same rates and watch the verdict flip to "not significant": identical percentages, a tenth of the evidence.

Significance is not importance — read the interval

A result can be statistically significant and commercially worthless: with a million visitors per variant, a lift of 0.02 percentage points will sail past p < 0.05 and still pay for nothing. The reverse also happens — a real, valuable effect reads "not significant" in a small test, which means only that the data cannot tell yet. That is what the confidence interval is for. It is the range of true effects compatible with what you observed: if it runs from −0.1 to +1.4 points, the honest summary is "somewhere between nothing and a solid win — collect more data or accept the uncertainty", not "no effect". Before shipping, weigh the interval against the cost of the change: a cheap change can be worth shipping on a promising-but-unproven interval; an expensive rebuild should wait until the interval's low end clears zero. And never confuse the two ways of stating a lift: +0.8 percentage points and +16% relative describe the same result above, and quoting the second without the first is how modest results get oversold in slide decks.

Sample size: the decision you make before the test

The planner above answers the only planning question that matters: how many visitors per variant do you need to detect the smallest lift you would act on? Four inputs drive it — your baseline rate, the minimum detectable effect (MDE), the confidence level, and the power, conventionally 80%: the probability the test catches an effect of that size if it is really there. The arithmetic is unforgiving. At a 5% baseline, detecting a 10% relative lift (5.0% → 5.5%) at 95% confidence and 80% power takes about 31,234 visitors per variant; detecting 20% takes about 8,158. Halving the effect you hunt roughly quadruples the traffic you need, which is why low-traffic sites should test bold changes — a new headline, a different offer — instead of button colours. Duration is then total visitors ÷ daily traffic, rounded up to whole weeks: Monday-morning visitors and Saturday-night visitors behave differently, and a test that stops mid-week has silently weighted one against the other. If the plan comes out longer than about eight weeks, the right move is usually a bigger change or more traffic to the page — not a smaller MDE typed in to make the plan look shorter.

The mistakes that quietly ruin tests

• Peeking. Checking every day and stopping at the first significant reading inflates a nominal 5% false-positive rate toward 30%. Fix the sample size in advance, then run to it — the peek guardrail above shows what repeated looks do to your verdict.
• Samples that are too small. With a few hundred visitors, only huge gaps reach significance, and a lift that does reach it in a tiny test systematically overstates the true effect (the winner's curse). The sensitivity readout shows the smallest lift your current traffic could have detected.
• Sample Ratio Mismatch. A planned 50/50 split that arrives 55/45 is not a curiosity — it signals a targeting, redirect or tracking bug, and it invalidates the comparison. The chi-square check above flags it; fix the machinery, restart the test.
• Too many variants, too many metrics. Ten independent comparisons at 95% confidence carry about a 40% chance that at least one looks significant by luck alone. That is why the Bonferroni correction is on by default here, and why you should name one primary metric before launch.
• Stopping a non-significant test late, hoping. Extending a test past its planned sample because the result is "trending" is peeking with extra steps. If the planned sample says no effect, the likeliest truth is that the change does little — redesign bolder.
• Novelty and seasonality. A new design often gets a brief curiosity bump (or dip) that fades; a test run across a sale, a holiday or a payday measures the calendar as much as the change. Run full weeks, ideally full business cycles, and be suspicious of effects that appear on day one.

Frequentist or Bayesian? You get both — here is why

The frequentist p-value answers "how surprising is this gap if nothing changed?" — the right question for a disciplined ship/no-ship call, and the language most teams already report in. The Bayesian chance-to-beat answers the question people actually ask — "how likely is B really better?" — by treating each conversion rate as a Beta distribution updated from a flat prior. Neither is the "true" statistics; they are two readings of the same evidence, and they occasionally disagree near the boundary, which is itself information: a result at p = 0.049 with a 91% chance-to-beat is a borderline call however you read it. This page also shows the expected loss of shipping each variant — the average conversion you give up, weighted by the probability you are wrong — because "significant" was never the real decision. The real decision is whether the expected gain beats the expected loss plus the cost of the change. When the test is one of several feeding a budget decision, pair it with the marketing ROI calculator; when the traffic comes from paid campaigns, the Meta ads calculator helps sanity-check the spend behind the numbers, and the automation ROI calculator applies the same show-your-formula discipline to workflow savings.