How to read A/B test results without guessing

Reading A/B test results well means checking three things before you trust the number: that the traffic split matches what you configured, that the result clears a significance threshold you set in advance rather than one you found by checking constantly, and that you picked the primary metric before the test started, not after. Skip any of the three and a result can look like a win and still be noise.

Most teams get this backwards. A dashboard turns a metric green, someone screenshots it for standup, and the test ships. Nobody asked whether the 51/49 split was supposed to be 50/50, whether that p-value was the first check or the fortieth, or whether “completed checkout” was the metric the hypothesis actually named. The test ran correctly. The reading of it didn’t.

This post covers the checks that separate a real result from a lucky-looking one: the traffic split, significance and the peeking problem, picking one metric in advance, segmenting without p-hacking a story, and closing the loop with a decision instead of a vibe.

Key Takeaways

• Check for a sample ratio mismatch (a chi-square test, flagged at p < 0.001) before trusting any number. An uneven split means the two groups were never comparable.
• Checking a p-value every day before the test ends inflates the false-positive rate well past 5%, the effect Evan Miller’s “How Not To Run An A/B Test” calls the peeking problem.
• A primary metric and its guardrails have to be named before the test starts. A metric chosen after you’ve already seen the data tells a story, and a story isn’t evidence.
• Segmenting a result to understand it is fine. Segmenting until one slice clears significance is the same bug as peeking, with more steps.
• dif conclude writes the decision and a one-line learning back to the surface file, so the result lives next to the hypothesis that produced it, not in a screenshot.

Check for a sample ratio mismatch before you trust the results

Before you read the metric, check the split. If you configured 50/50 and 100,000 exposures land at 51,000/49,000, that gap is small enough to shrug off and still fails a chi-square goodness-of-fit test at p < 0.001, the threshold most teams use because this check runs on every experiment they ship.

A mismatch almost never means the randomization math is broken. It usually means a caching layer served one variant without a fresh assignment, or an exposure event quietly failed to fire for a slice of traffic, which makes the two groups different populations before anyone looks at the outcome metric. Sample ratio mismatch covers the chi-square test and the usual causes in full. Run it first. If it fails, nothing past this point in the post matters yet.

Statistical significance in A/B test results isn’t a green checkmark

A p-value answers one narrow question: if there were truly no difference between variants, how often would a gap this large show up by chance? p < 0.05 means that would happen less than 5% of the time, nothing more.

It says nothing about the size of the effect, or whether that size matters to the business. A confidence interval does more work here: a 95% CI of [0.3%, 4.1%] on checkout completion is a real lift, but a wide interval that barely clears zero is a different result than a tight one centered on 2%, even if both cross the same significance line. The NIST/SEMATECH handbook’s section on confidence limits is a clean reference if the mechanics are rusty. Read the interval, not just the pass/fail.

The peeking problem inflates your false positive rate

Checking a test’s p-value once isn’t the same as checking it daily until it crosses 0.05. Every additional look is another chance for noise to cross the line, and the effective false-positive rate climbs well past the 5% you set when you decided on that threshold once, at the start.

Evan Miller’s “How Not To Run An A/B Test” names this the peeking problem and is the canonical write-up. The fix is deciding your stopping rule before you launch, not after: a fixed sample size, or a fixed calendar window, written into the experiment’s brief alongside the hypothesis.

That costs you something real. If day three looks terrible, you still wait out the window you set, unless a guardrail metric breaks, because reacting to an early number is exactly the behavior that produces false positives in the first place. Dashboards that show a live significance number every day invite exactly that reaction; use them to watch for a sample ratio mismatch or a broken guardrail, not to decide when the test is done.

Pick one primary metric before you read the results

An experiment needs a primary metric and one or two guardrails, named in the frontmatter before the test runs:

metrics:
  primary: completed_checkout
  guardrails:
    - refund_rate
    - page_load_time

The primary metric is the one the hypothesis is actually about. Guardrails exist to catch a win that costs more than it earns, a copy change that lifts checkout but also lifts refunds. Trustworthy Online Controlled Experiments (Kohavi, Tang, and Xu) is the standard reference for why an overall evaluation criterion beats checking a dozen metrics and reporting whichever one moved. If a secondary metric looks interesting after the fact, that’s a hypothesis for the next test. It isn’t a result for this one.

Segment results without p-hacking a story

Segmenting a result by device, country, or traffic source is a reasonable way to understand why a number moved. Slicing the same data by ten different attributes until one segment clears significance is a different activity: run enough cuts and one will cross p < 0.05 by chance alone, even with zero real effect anywhere.

If a segment looks different, that’s the start of a new hypothesis. It isn’t the conclusion of this one. For a specific user, dif qa --user u_8131 --attr device_type=mobile traces exactly what variant they got and why, which is useful for debugging a suspicious segment without re-running a statistical test on an ever-smaller slice. The CLI reference covers the full qa command.

Conclude with a decision, not a vibe

Once the split checks out, the significance held without peeking, and the primary metric is the one the hypothesis named, the result is a decision: ship it, hold it, or kill it. dif conclude dates that decision and writes a one-line learning back to the surface file the experiment belonged to, so the next person opening that surface reads what was already tried instead of re-running it.

That’s the difference between a result and a screenshot. A screenshot is a number someone believed on a Tuesday. A concluded experiment is a file with the hypothesis, the metric, the decision, and the reason, all in one place, reviewable the same way the code was. The A/B test lifecycle covers how a dif experiment moves from draft to conclusion, and experimentation in git covers the format if you haven’t set one up.

FAQ

How do you know if A/B test results are statistically significant? Check the p-value against the threshold you set before the test started, usually 0.05, and look at the confidence interval, not just the pass/fail. A result that clears significance with a wide interval straddling a small effect is weaker evidence than a tight interval with the same p-value.

How long should you wait before reading A/B test results? As long as the fixed sample size or calendar window you set before launch, whichever you chose. Checking the result early and stopping the moment it looks significant is the peeking problem, and it inflates the false-positive rate past the threshold you think you’re using.

What is a sample ratio mismatch in A/B testing? It’s when the observed split across variants differs from the configured weights by more than chance explains, usually flagged with a chi-square test at p < 0.001. It almost always traces to a caching or logging bug, not the randomization itself, and it has to be ruled out before the metric means anything.

Why do my A/B test results keep changing day to day? Early results on a small sample are noisy, and a metric that looks significant on day three can regress to nothing by day ten. That instability is the peeking problem showing up visually: the fix is a predetermined sample size or duration, not watching the number more closely.

What’s the difference between statistical and practical significance? Statistical significance means the result is unlikely to be chance. Practical significance means the effect is large enough to matter. A 0.1% lift on a huge sample can be statistically significant and still not worth shipping; check the confidence interval’s actual magnitude, not just whether it excludes zero.

Conclusion

Reading A/B test results well is three checks, done in order: the traffic split, significance without peeking, and a primary metric chosen in advance. A chi-square test on exposure counts catches a mismatch before it wastes anyone’s time on the metric. A confidence interval says more than a p-value alone. And a decision only counts once it’s written down next to the hypothesis that produced it, not screenshotted and forgotten.

If you’re running experiments in dif, the hypothesis, metrics, and guardrails live in the same frontmatter the SDK reads, and dif conclude closes the loop. npm install -g @dif.sh/cli followed by dif init gets a working experiment set up in about two minutes, and you don’t need an account to start.