Sample ratio mismatch: what it means and how to catch it

A sample ratio mismatch (SRM) is when the observed split of users across an A/B test’s variants differs from the ratio you configured, by more than random chance explains. Set a 50/50 split and watch 100,000 users land 51,000 to 49,000 instead of close to even, and that gap is probably a bug, not noise.

The bug is rarely in the randomization math. A caching layer serves one variant’s page without a fresh assignment, or an exposure event quietly never fires for a slice of browsers, and the two groups stop being comparable before anyone looks at the outcome metric. By the time someone notices the primary metric moved, the groups were already different populations.

This post covers what a sample ratio mismatch is, the chi-square test that catches it, the usual causes, and what to do once you have found one.

Key Takeaways

A sample ratio mismatch is when an experiment’s observed variant split deviates from the configured weights by more than chance, typically flagged at p < 0.001 on a chi-square test.

SRM invalidates the comparison before you look at the metric. If the groups were not split the way you intended, a measured lift could be selection, not the change.

Most SRMs trace to bot traffic, a caching layer, or an exposure event that fails to fire, not to a broken randomization algorithm.

A chi-square goodness-of-fit test on exposure counts alone catches it, with no access to the outcome metric required.

Deterministic hashing removes the “a refresh reassigns you” class of SRM, but a CDN cache or a client-side exception before the exposure call still produces one.

What a sample ratio mismatch looks like

Say you run an experiment at a 50/50 split and collect 100,000 exposure events: 50,000 expected per variant. The groups actually land at 51,000 and 49,000, a 2% gap. That looks small enough to shrug off, but at this sample size it is not what chance produces on its own.

The test compares observed counts to expected counts: chi-square equals the sum of (observed minus expected) squared, divided by expected, across each group. For two variants that is one degree of freedom, and a chi-square value above 10.83 corresponds to p < 0.001.

For the 51,000/49,000 split: (1,000² / 50,000) × 2 = 40. That clears the 10.83 threshold by a wide margin, which means this 2% gap is not something you’d expect from a healthy coin flip at that sample size. Something is systematically routing users unevenly.

Catch a sample ratio mismatch with a chi-square test

Most teams use p < 0.001 as the flag, stricter than the usual 0.05. Microsoft’s experimentation research group recommends the tighter bar because this check runs on every single experiment, and a 5% false-positive rate would flag a healthy test constantly.

Running it takes one function call against the exposure counts your tracking sink already has:

from scipy.stats import chisquare

observed = [49000, 51000]
expected = [50000, 50000]
chi2, p = chisquare(observed, f_exp=expected)
# chi2 = 40.0, p < 0.0001 -> flag it

scipy.stats.chisquare is a goodness-of-fit test, documented in the scipy reference, and the underlying math is the standard chi-squared test. Run it on the exposure counts before you touch the primary metric. If it fails, the metric comparison is not trustworthy yet.

The usual causes aren’t bad randomization code

A sample ratio mismatch almost always traces to infrastructure around the experiment, not the bucketing math itself:

Bot and crawler traffic lands disproportionately in one variant because it hits a cached or default path before anything assigns a bucket.
A CDN or service worker cache serves one variant’s rendered page repeatedly, without a fresh assignment or a fresh exposure event behind it.
An exposure event never fires for a slice of users, often because the new variant’s code path throws before reaching the tracking call.
A redirect-based test adds latency for one variant, and slow connections time out before the redirect completes, so those users never get counted.
An audience or exclusion filter that is supposed to be symmetric across variants is not, so one variant’s eligible population is quietly smaller.
Hashing on the wrong id, like a session id instead of a user id, lets a page refresh reassign someone and skews whichever bucket fills first.

Where deterministic bucketing still lets one through

Deterministic, hash-based bucketing removes the last cause structurally. dif assigns a variant from the first four bytes of SHA-256(salt || user_id) mod 10,000, recomputed identically every time. Nothing is stored and nothing moves on a refresh, so the “a refresh reassigns you” class of SRM cannot happen.

What it does not fix is what sits in front of the correct assignment. A CDN caching an SSR’d page by URL instead of by user can keep serving yesterday’s variant to today’s request, even though the hash computed it correctly on the server. The bucket was right; what got served was not.

It also does not fix logging. Exposure fires once per flag, per user, per session, at render time. If the new variant’s component throws before that line runs, the user was assigned correctly but never recorded, and your exposure counts look imbalanced even though the real split underneath is fine. That is a logging SRM, not a randomization SRM, and the chi-square test cannot tell you which you have. You have to go look.

Every failure mode in dif resolves to the first variant and records nothing, so a bug only ever costs you “nobody saw the new thing,” not a bad incident. The same property means that failure is invisible in the exposure count. Compare exposure counts against total eligible traffic for the surface, not just the two variants against each other, or a silent fail-open can hide underneath a chi-square test that technically passes.

Exclusion groups are the other place to check. Two experiments sharing an exclusion_group are ordered by created date, and the first matching audience wins while the rest show their default and record nothing. If the audiences were supposed to be disjoint and aren’t, traffic bleeds from one experiment into the other, and it shows up as SRM in both. The format reference covers how exclusion groups and audiences are declared.

What to do once you have caught one

Pause the test. Looking at the primary metric while SRM is present is how a composition effect gets mistaken for a real lift.

Segment the imbalance by device, browser, country, and traffic source. If it concentrates in one browser or one region, that is usually a caching or bot signature, not a global bug. The A/B test lifecycle covers how exposure and tracking fit together, which is the part worth re-checking first.

Trace a specific affected user instead of guessing. dif qa --user u_8131 --attr device_type=mobile shows exactly what variant that user was assigned and why, which you can compare against what actually got served or logged. The CLI reference covers the full qa command.

Check whether an audience or exclusion condition changed recently. Git is the audit log here: a git log on the experiment file shows exactly when a weight or audience changed, which usually lines up with when the imbalance started.

Fix the root cause before you re-run. Extending the test or re-bucketing without fixing the underlying bug just reproduces the same mismatch at a larger sample size.

FAQ

What causes a sample ratio mismatch? Most SRMs come from instrumentation and infrastructure, not broken randomization code: bot traffic, a caching layer serving a stale variant, an exposure event that silently fails to fire, or an audience filter that is not actually symmetric across variants.

How do you test for sample ratio mismatch? Run a chi-square goodness-of-fit test comparing each variant’s observed exposure count against what the configured weights predicted. A chi-square value above 10.83 for a two-variant test corresponds to p < 0.001, the threshold most teams use to flag it.

What p-value should flag a sample ratio mismatch? p < 0.001 is the common bar, stricter than the usual 0.05 significance level. The check runs on every experiment a team ships, so a looser threshold would flag healthy tests constantly.

Does a sample ratio mismatch mean the A/B test is invalid? Yes, for the metric comparison, until you explain it. If the groups were not split the way you intended, any difference in the outcome metric could be a composition effect rather than the change you shipped.

Can a feature flag tool cause a sample ratio mismatch? Yes, through the infrastructure around it rather than the bucketing math. A CDN cache serving a stale variant, an exposure event that never fires because of a client-side error, or an exclusion group with overlapping audiences can all produce one even when the underlying hash assignment is correct.

Conclusion

A sample ratio mismatch is a traffic-split problem, not a metrics problem, and it has to be ruled out before the metric means anything. Run the chi-square test on exposure counts first. Most causes trace to bots, caching, or a tracking call that never fired, not to the randomization itself. Deterministic hashing closes off the refresh-reassignment case, but a cache or a logging gap in front of it can still produce one.

If you’re running experiments in dif, exposure counts live wherever your tracking sink writes them, dif Cloud or your own custom handler, so that is where to run the check. If you haven’t set one up yet, experiments as files covers the format, and npm install -g @dif.sh/cli followed by dif init gets you a working one in about two minutes.