A/B test statistical significance, explained
Statistical significance in an A/B test means the gap between control and variant is unlikely to be random noise, at whatever threshold you set in advance (usually p < 0.05). It does not mean the lift is large, that it matters to the business, or that you are 95% certain the variant is better. Those are three separate, common misreadings of the same number.
Most teams treat “significant” as a finish line: the dashboard turns green, someone screenshots it, the test ships. The number behind that green checkmark, the p-value, answers a narrower question than the one people think it answers, and the gap between those two questions is where a lot of bad ship decisions come from.
This post covers what statistical significance measures, the two misreadings that cause the most damage (the p-value’s real meaning, and what 95% confidence does not mean), why checking the result early inflates your false positive rate, and where the significance check happens in a dif experiment.
Key Takeaways
- Statistical significance means the observed gap is unlikely under the assumption that control and variant perform identically, at a threshold set before the test, not that the lift is real, large, or worth shipping.
- A p-value of 0.03 means: if there were truly no difference, you’d see a gap this large or larger about 3% of the time. It is not the probability the variant is better.
- Checking a test for significance every day instead of once at the planned sample size can push the real false positive rate well above 5%, a problem known as peeking.
- Significance and effect size are different questions. A tiny, real lift can be significant on enough traffic; a large lift can be insignificant on too little.
- dif Cloud computes significance on your exposure and track events and writes the result back to your PR as a comment, but the decision to ship still belongs to a person reading the number in context.
What statistical significance means
Statistical significance is the answer to one specific question: assuming control and variant perform identically (the null hypothesis), how surprising is the gap you observed? If it’s surprising enough, below your significance level, usually 5%, you call the result significant and reject the assumption that they’re the same.
That’s it. It’s a test against a straw-man assumption of “no difference,” not a measurement of how good the variant is. A result can be significant and still be a 0.4% lift that nobody would notice. A result can be an eye-catching 20% lift and still not be significant, if the sample was small.
The p-value answers a narrower question than people think
A p-value of 0.03 is routinely read as “there’s a 3% chance this result is wrong” or “we’re 97% sure the variant wins.” Neither is what it says.
The correct reading: if control and variant truly performed identically, a gap this large or larger would show up about 3% of the time from sampling noise alone. The p-value describes how unlikely your data is under an assumption of no effect. It says nothing directly about the probability that the variant is better; that question requires a different kind of math (Bayesian, with a prior) that most A/B testing tools don’t run.
The ASA statement on p-values puts it plainly: a p-value does not measure the probability that the studied hypothesis is true, or the probability that the data were produced by random chance alone. It measures how incompatible the data are with a specific statistical model, the one where control and variant are the same.
95% confidence doesn’t mean 95% chance you’re right
The companion misread sits on the confidence level. “We ran it at 95% confidence” sounds like “there’s a 95% chance the variant is better.” What 95% confidence describes is the long-run behavior of the method: if you ran the same experiment over and over with no real difference between variants, about 95% of those runs would correctly fail to flag a significant result, and about 5% would falsely flag one anyway. That 5% is the significance level, chosen before the test, not a probability statement about this one result.
The NIST/SEMATECH Engineering Statistics Handbook lays out the two-proportion test most A/B tools run under the hood: compare the two observed rates, account for sample size, and produce the p-value that gets compared against the significance level. The math is mechanical. The interpretation is where people go wrong.
Peeking inflates your false positive rate
A test’s p-value moves around while the test is running, dipping in and out of “significant” before settling down. Check it every day and act the first time it crosses 0.05, and you are no longer running one test at a 5% false positive rate. You’re running many tests, one per peek, and taking the first one that clears the bar.
Evan Miller’s write-up on how not to run an A/B test shows the scale of the problem: checking daily for two weeks can push the real false positive rate well past 5%, sometimes over 20%, even when there is no true difference at all. The fix is deciding the sample size or the duration before the first exposure, the same number covered in minimum detectable effect, and checking once the test reaches it. If you need to monitor a test for safety, watch a guardrail metric for a clear regression, not the primary metric’s p-value, and don’t call the result until the planned sample size is in.
Significant isn’t the same as big, or worth shipping
Significance is a yes-or-no answer to “is this gap probably not noise.” It says nothing about whether the gap is large enough to matter, or worth the cost of shipping and maintaining the winning variant.
On enough traffic, a real but tiny lift, say 0.3 percentage points on a metric with millions of exposures, will clear p < 0.05 easily. It’s statistically significant and might be commercially irrelevant. The reverse happens constantly on low-traffic surfaces: a real 15% lift sits at p = 0.11 because the sample never got big enough to resolve it, and the test gets called a loss. Pairing significance with a primary metric and a guardrail, decided before the test starts, catches the second failure mode: a significant win on the primary that hides a significant loss somewhere else.
---
id: checkout-cta-copy
status: active
owner: sam@acme.com
surface: checkout
metrics:
primary: completed_checkout
guardrails:
- refund_rate
---
That shape, one primary metric plus at least one guardrail, is what keeps “significant” from meaning “we found a number we liked and stopped looking.”
Where the significance check happens in dif
An A/B test in dif is a Markdown file with a hypothesis, a split, and a primary metric, the same shape covered in A/B testing for developers. If you run dif Cloud, it ingests the exposure and track events as they arrive and computes significance on them, writing the result back to your pull request as a comment:
dif Cloud: completed_checkout
control 3.0% (12,400 exposures)
get_it_today 3.4% (12,380 exposures)
relative lift +13.3%
p-value 0.031
significant yes (p < 0.05)
That number lands in the same place the flag weights and the owner get reviewed, not a separate dashboard someone has to remember to check. It still answers the same narrow question: the gap is unlikely to be noise at the threshold you set. Whether 13.3% on completed_checkout is worth the engineering cost of keeping get_it_today around is a judgment call for the person reading the PR, not something the p-value decides for them.
FAQ
What does statistical significance mean in A/B testing? It means the observed difference between control and variant is unlikely to have happened by chance alone, at a threshold (commonly p < 0.05) set before the test started. It does not mean the lift is large or worth shipping.
What is a good p-value for an A/B test? The standard threshold is 0.05 (a 5% significance level), chosen before the test, not after seeing the data. Some high-stakes tests use a stricter threshold like 0.01; the right choice depends on the cost of a false positive for that specific change.
Does statistical significance mean my result is important? No. Significance says the gap probably isn’t noise. It says nothing about effect size or business value. A tiny, real lift can be significant with enough traffic, and a meaningful lift can fail to reach significance on too little traffic.
Why did my test look significant, then stop being significant? The p-value fluctuates while a test runs. Checking it repeatedly and stopping the first time it crosses 0.05 (peeking) inflates your real false positive rate well above the 5% you think you’re running at. Decide the sample size up front and check once you reach it.
What’s the difference between statistical significance and minimum detectable effect? Minimum detectable effect is set before the test to decide how much traffic you need; statistical significance is computed after the test to decide whether the observed gap is likely real. One is a plan, the other is a result. Minimum detectable effect covers the planning side in full.
Takeaways and next step
Statistical significance answers one question: is this gap unlikely to be noise, at a threshold you set before looking. It is not a measure of size, importance, or certainty that the variant is better, and checking it daily instead of once at the planned sample size quietly breaks the 5% false positive rate you think you’re running. Pair it with a primary metric and a guardrail decided in advance, and read the number in the context of what shipping the change actually costs.
If you want that hypothesis, split, and significance result to live in the same file as the flag weights, reviewed in the same pull request:
npm install -g @dif.sh/cli
dif init
dif’s experimentation workflow keeps the plan and the result in the repo, and the first experiment guide walks through one end to end.