Minimum detectable effect: how big a lift you can see
Minimum detectable effect (MDE) is the smallest lift an A/B test is powered to detect, given its sample size, baseline rate, and significance level. Set the MDE too small for the traffic you have and the test needs more sessions than the surface will ever see. Set it too big and a real, smaller win reads as “no difference.”
Most teams skip this step. They pick a split, a duration that feels reasonable, two weeks is a common default, and run it the way an A/B test usually runs in code. Two weeks on a surface with 600 sessions a day might be powered to detect a 40% relative lift. If the real change moved the metric by 8%, the test was never going to see it, and “not significant” gets reported as “the change did not work,” which is a different claim than the data supports.
This post covers the formula behind minimum detectable effect and what it costs in real sample size. It also covers why a guardrail metric usually needs a different MDE than your primary metric, and where to write the number down before you ship.
Key Takeaways
- Minimum detectable effect, baseline rate, and sample size are locked together by one formula. Fix any two and the third is decided for you.
- On a 3% baseline conversion rate, detecting a 10% relative lift takes roughly 51,700 sessions per variant. Detecting a 2x lift takes about 520.
- The relationship is quadratic: halving the MDE roughly quadruples the sample size you need, not doubles it.
- Decide the MDE before the first exposure and write the target sample size into the experiment’s brief, the same file that holds the hypothesis.
- A guardrail metric usually needs a larger MDE than the primary metric, because the goal is catching a real regression, not detecting a small win.
What minimum detectable effect means
Minimum detectable effect is the smallest true difference between control and variant that your test has a good chance of catching. “Good chance” usually means 80% power: run the same test 100 times at that true lift, and about 80 of them come back significant.
MDE is not something you observe after a test. It is something you set before one starts, alongside the baseline rate and the significance level, to answer a different question: how much traffic do I need to run this for? Reading a result without having set an MDE first is how an underpowered test gets misread as a flat one.
The three numbers that decide your sample size
Hold significance and power at their usual defaults (5% two-sided, 80% power) and three numbers move together: baseline rate, minimum detectable effect, and sample size. Fix any two and the third is decided for you.
For a conversion-rate metric, the standard approximation at those defaults is:
n (per variant) ≈ 16 × p × (1 − p) / d²
where p is the baseline rate and d is the absolute lift you want to detect. It is an approximation, not an exact value. Evan Miller’s sample size calculator runs the precise version if you want exact numbers for your rate, significance level, and power.
Why a smaller minimum detectable effect costs more traffic
The lift is squared in the denominator, so the cost of precision grows fast. Start from a 3% baseline conversion rate:
| Target lift | New rate | Absolute d | Sessions per variant |
|---|---|---|---|
| 10% relative | 3.3% | 0.003 | ~51,700 |
| 2x (100% relative) | 6.0% | 0.03 | ~520 |
Moving the target lift from a 2x change to a 10% change does not cost 10x the traffic. It costs 100x, because the lift is squared. The same math on a higher-traffic, higher-baseline metric looks friendlier: at a 20% baseline rate, detecting a 10% relative lift needs about 6,400 sessions per variant, and detecting a 20% relative lift needs about 1,600. Low-baseline, low-lift tests are where teams run out of patience before they run out of theory.
Write the MDE into the experiment before you ship
In dif, the hypothesis and the plan live in the same Markdown file as the flag weights. The sample size target doesn’t have to live in one person’s head. Add it to the brief, since the brief is free text, rather than inventing a new frontmatter field:
---
id: pricing-cta-copy
status: active
owner: sam@acme.com
surface: pricing
hypothesis: >
A benefit-led CTA ("See your plan") lifts completed_checkout
over the control ("Upgrade now").
variants:
- id: control
weight: 50
- id: see_your_plan
weight: 50
metrics:
primary: completed_checkout
guardrails:
- refund_rate
---
## Brief
Baseline completed_checkout is 3%. Target a 10% relative lift
(3.0% -> 3.3%), which needs ~51,700 sessions per variant at 80%
power. Hold at 50/50 until each variant clears that count, then
conclude. Do not call it before then.
That block is reviewed in the pull request that ships the flag, the same place the weights and the owner get checked. The .md format covers the rest of the frontmatter; the planning number is prose, written once, read by whoever looks at the file next.
If you run dif Cloud, it computes significance on the exposure and track events once they arrive, and writes the result back to your PR. It has no idea what MDE you planned around. That number lives in the brief, not in a stats engine, because you decide it before the first user is exposed, not after the data comes in.
Guardrails need their own MDE, not the primary metric’s
The primary metric and the guardrail answer different questions, so they should not share one MDE. The primary metric is “is there a real win worth this size,” and a tight MDE is the point. A guardrail metric is “did this make something meaningfully worse,” and the bar is usually a bigger MDE, because a guardrail miss that is small does not change the ship decision the way a guardrail miss that is large does.
In the example above, refund_rate almost certainly has a lower baseline and fewer events than completed_checkout. It reaches statistical power far more slowly for the same lift size. Powering the guardrail for a 10% MDE when the primary is also at 10% usually means the guardrail never finishes. Set a guardrail MDE that matches the size of regression you actually need to catch, often a 20-30% relative move, not the primary metric’s number.
When you skip this, you get a false “no difference”
An underpowered test and a test with no real effect produce the same headline: “not significant.” The difference only shows up if someone asks what MDE the test was actually powered for. Evan Miller’s write-up on common A/B testing mistakes covers the related failure of stopping a test the moment it looks significant. An underpowered test is the quieter version of that same problem. Nobody stops it early; it just runs out the clock, and the sample size error only surfaces if someone goes looking for it.
FAQ
What is a good minimum detectable effect for an A/B test? There is no universal number. It depends on your baseline rate and your traffic: pick the smallest lift that still matters to the business and check whether your traffic can reach that sample size in a reasonable window. If it cannot, either accept a larger MDE or run the test longer.
How do I calculate minimum detectable effect? Work backward from the traffic you actually have and your test duration to find the sample size per variant, then solve n ≈ 16 × p(1-p) / d² for d given your baseline rate p. Optimizely’s glossary entry and the calculator above both make this a lookup instead of algebra.
Does a smaller MDE always mean a better test? No. A smaller MDE means a more sensitive test, but sensitivity costs traffic, and the relationship is quadratic. A smaller MDE than your real decision needs just burns sessions you could have spent on a different test.
What’s the difference between MDE and statistical significance? MDE is set before the test to decide the sample size; significance is computed after the test to decide whether the observed difference is likely real. One is a plan, the other is a result.
Should a guardrail metric use the same MDE as the primary metric? Usually not. The primary metric is powered to catch a small win; a guardrail is powered to catch a real regression, which is normally a larger, looser MDE so the guardrail reaches power on the same timeline as the primary.
Takeaways and next step
Minimum detectable effect ties your sample size, baseline rate, and significance level into one equation, and the lift is squared in that equation, so chasing a smaller MDE gets expensive fast. Decide the MDE before the first exposure, size the guardrail separately, and write the target sample size into the experiment file next to the hypothesis, not in a Slack thread.
If you want that file to also hold the flag weights, the owner, and the decision when you conclude, run your first experiment or install the CLI:
npm install -g @dif.sh/cli
dif init
dif’s experimentation workflow keeps the plan, the split, and the result in the same repo as the code it tests.