Feature flag vs A/B test: what is actually different
A feature flag and an A/B test are the same mechanism pointed at different questions. Both wrap a code path in a condition and pick which version a given user sees. The difference is intent and what ends it. A feature flag is a switch you plan to ramp to 100% and then delete. An A/B test is that same switch held at a fixed split until a metric answers a hypothesis.
Teams get this wrong in a recognizable way. Someone runs a copy test in the release-flag tool, ships the winner, and six months later nobody can say what the number was, because there was never a field to write it in. Or a flag sits at 50/50 with no hypothesis and no owner for eight months while two teams argue about which system owns it.
This post is about where the two diverge and which one to reach for when a change could plausibly be either.
Key Takeaways
- A feature flag and an A/B test run on one mechanism: a per-user assignment that picks a code path. What differs is the weights, whether there is a hypothesis, and what ends it.
- A flag ends at 100% with the losing branch deleted. An A/B test ends when the primary metric clears significance and someone records a decision.
- In dif both are one Markdown file under
dif/experiments/active/, sharing the samevariants,metrics, and validator. A 90/10 rollout and a 50/50 test differ by two numbers.- An A/B test costs traffic a rollout does not. Moving a 5% conversion rate to 5.5% takes roughly 30,000 users per variant at 80% power, which is seven months at 2,000 sessions a week.
- Reach for a flag when you have already decided to ship. Reach for an A/B test when the decision depends on the number.
Feature flag vs A/B test: the difference is intent
Both are a runtime condition around a code path, and both assign each user to one side of it. What separates a feature flag from an A/B test is what you plan to do with the result.
A feature flag controls a release. You decided the new checkout is better, and the flag is how you turn it on for 1%, then 10%, then everyone, with a way back if the error rate climbs. The end state is 100% and a deleted branch.
An A/B test answers a question. You hold two versions at a fixed split, usually 50/50, because you do not know which one wins. The end state is a number and a decision written down.
Everything under that is identical: the assignment math, the exposure event, the call at the render site. The feature toggle taxonomy on martinfowler.com makes the same cut, separating release toggles from experiment toggles by how long they live and who changes them, not by how they work.
Same file, different frontmatter
In dif, a rollout and a test are two files in the same directory, sharing one .md format. Here is the rollout, at dif/experiments/active/new-checkout.md:
---
id: new-checkout
status: active
owner: ada@acme.dev
surface: checkout
hypothesis: >
The rewritten checkout holds completed_checkout flat while cutting
errors on the payment step. Ramping, not testing.
variants:
- id: "off"
weight: 90
- id: "on"
weight: 10
metrics:
primary: completed_checkout
guardrails: [payment_error_rate]
created: 2026-09-14
---
## Brief
Ramp to 25% once payment_error_rate holds for a week, then 50, then 100.
And the test, at dif/experiments/active/checkout-cta-v2.md:
---
id: checkout-cta-v2
status: active
owner: ada@acme.dev
surface: checkout
hypothesis: >
A benefit-led CTA ("Get it today") lifts completed_checkout over the
control ("Place order") for returning visitors.
variants:
- id: control
weight: 50
- id: variant_a
weight: 50
summary: '"Get it today" copy'
metrics:
primary: completed_checkout
guardrails: [refund_rate]
exclusion_group: checkout-copy
created: 2026-09-14
---
## Brief
Hold at 50/50 for two weeks minimum. Conclude when completed_checkout
clears significance and refund_rate holds.
Two files, one schema. The rollout differs in its weights, in the exclusion_group the test declares and the rollout does not, and in what the brief says happens next. Both pass the same dif validate, compile through the same dif build, and are read at the render site by the same call:
if (dif("checkout-cta-v2") === "variant_a") {
return <Cta copy="Get it today" />;
}
return <Cta copy="Place order" />;
Application code never learns whether it is gating a rollout or a test. That is why a test can become a rollout later without a code change. The format requires a hypothesis on every file, rollouts included. On a pure release flag that reads like paperwork. It is also the reason a file nobody has touched in six months still says why it exists.
What changes between a feature flag and an A/B test
| Dimension | Feature flag | A/B test |
|---|---|---|
| Question it answers | Is it safe to turn this on? | Which version moves the metric? |
| Weights | Moving: 1, then 10, then 50, then 100 | Held: usually 50/50 |
| Hypothesis | Optional in most tools, and thin when present | Load-bearing; the test is meaningless without one |
| Primary metric | A guardrail, watched for regressions | The thing being measured |
| Typical lifetime | Days to a few weeks | Two weeks to a quarter, set by sample size |
| What ends it | 100% and a deleted branch | Significance, then a recorded decision |
| Traffic needed | None in particular | Enough to detect the effect you care about |
| Failure mode | Stuck at 100% for three years with dead code behind it | Concluded in a Slack thread, then re-run in two years |
The row that catches teams out is traffic. A rollout works at any volume, because you are watching for breakage, not measuring a lift. A test at the same volume can run for a quarter and still return noise.
When a feature flag is the right call
Use a flag when the decision is already made and the open question is risk. Concretely:
- You decided to ship it. A rewritten checkout, or a search backend you already migrated to in staging. You want a staged rollout and a way back, not a verdict.
- There is no metric to move. A refactor, an infrastructure swap, a dependency upgrade. Nothing user-facing changes, so there is nothing to measure.
- You need a kill switch. The flag exists so someone can turn the thing off at 3am without reverting a week of work.
- Traffic is too thin to measure. If the numbers cannot answer, ramp carefully and watch guardrails instead of pretending a test is running.
- You are dark launching. The code ships at 0% and stays off until the rest of the work lands.
For a PM, the useful test is one question: if the metric came back flat, would you still ship it? If yes, you want feature flags in your repo, not an experiment.
When to run an A/B test instead
Run a test when the decision depends on the number, you can name one primary metric before you start, and you have the traffic to detect the effect.
That last condition is the one people skip. The rule of thumb for sample size is n = 16 * p * (1 - p) / d^2 per variant, at 80% power and 5% significance, where p is the baseline rate and d is the absolute lift you want to detect. Take a 5% conversion rate and a target of 5.5%:
n = 16 * 0.05 * 0.95 / (0.005 * 0.005)
= 30,400 users per variant
= 60,800 total
At 2,000 checkout sessions a week, that is about seven months. Evan Miller’s sample size calculator gives the exact figure for your baseline and effect size. Run it before you write the hypothesis, not after week three, because the answer often tells you to ship behind a flag instead.
Test anyway when the change is cheap to try and expensive to be wrong about. Ronny Kohavi and his colleagues at Microsoft’s experimentation platform have published that roughly a third of tested ideas move the metric they were designed to move. That is the argument for measuring, and also why a test you cannot power is worse than no test at all: you get a number, it means nothing, and somebody ships on it.
Ramp the winner, and the cost of doing it in git
An A/B test that wins becomes a flag you ramp. In dif that is one command and one edit, and no change at the call site.
dif conclude checkout-cta-v2 \
--decision "Shipped variant_a. +2.1% completed_checkout over 14d, refund_rate flat."
dif conclude moves the file to dif/experiments/concluded/2026-09-checkout-cta-v2.md, stamps status: concluded with today’s date, fills the ## Decision block, and appends one dated line to dif/surfaces/checkout.md:
## Learnings
- 2026-09-28 checkout-cta-v2: shipped variant_a. +2.1% completed_checkout over 14d, refund_rate flat.
- 2026-08-02 trust-badges-row: no effect on conversion.
The next dif new on that surface reads the last three learnings into the draft, so the next person to propose a checkout copy test starts from what already happened. A dashboard row has no field for that sentence, and deleting the row takes the result with it. The surface file keeps the line after the experiment file is archived. Ramping the winner is then editing weight: 50 to weight: 100 on the shipping variant and opening a PR, which the dif CLI validates in CI before it merges.
Here is what that costs. Every change is a merge and a deploy, including the rollback. If your build takes ten minutes, your worst-case kill is ten minutes, where a dashboard toggle is a second. For a payment path that needs a sub-second kill, use something built for that on that one flag. For most rollouts, where the failure mode is that some users see the new thing, a merge and a deploy is fine, and the reason you rolled back is in the git history instead of in someone’s memory.
FAQ
Is an A/B test just a feature flag?
Mechanically, yes. An A/B test is a flag held at a fixed split with a hypothesis and a primary metric attached. The same assignment function decides both. What makes it an experiment is that you agreed in advance what number would settle the question, and that you write the answer down when it comes.
Can you use feature flags for A/B testing?
Yes, and most teams do. A flag gives you the split and the per-user assignment, which is the hard half. What a plain flag tool usually does not give you is a hypothesis field, a declared primary metric, and a decision that survives the flag’s deletion. If your flag tool has nowhere to record the result, budget for recording it somewhere that will still be readable in a year.
When should I use a feature flag instead of an A/B test?
When you would ship the change even if the metric came back flat. Refactors, migrations, infrastructure swaps, and anything with no user-facing metric are flags. So is any change on traffic too low to power a test, where a careful ramp with guardrails beats a result you cannot trust.
Do I need two tools for feature flags and A/B testing?
No. Every capability an A/B test needs (deterministic assignment, held weights, exposure events, a primary metric) is a superset of what a flag needs, so one system can do both. In dif they are the same Markdown file under dif/experiments/active/, and the difference is the weights and whether you are holding or ramping. Running experiments as files means a flag becomes a test, or a test becomes a flag, by editing frontmatter.
How long should an A/B test run?
Long enough to reach the sample size your effect size requires, and at least one full week to cover the weekday and weekend cycle. Two weeks is the common floor. Calculate the number first; stopping the moment a result looks significant inflates false positives, which is how a team ships three wins in a row that never show up in the quarterly numbers.
Pick the one that matches the decision
Feature flag vs A/B test comes down to four things. A flag controls a release and ends at 100%. A test answers a question and ends at a decision. A test needs traffic a flag does not, so run the sample size math before you write the hypothesis. And both run on the same assignment mechanism, so one file format can carry either.
If they live in the same place, the switch between them costs nothing. A test that wins becomes a flag you ramp by editing a weight, and the result stays attached to the surface it ran on instead of expiring with a Slack thread.
To try it, install the CLI and scaffold a workspace:
npm install -g @dif.sh/cli
dif init
Then walk your first experiment end to end, from dif new to dif conclude. There is no account and no API key, and the file it writes is the same one you would use for a rollout.