Get started
← Blog

What is a holdout experiment? When to run one

Sep 15, 2026

A holdout experiment keeps a small group of users on the old experience while everyone else gets the new one, then measures the gap over weeks or months. The holdout group is a control that outlives the test that created it, which is how you find out what a shipped change actually did.

Teams usually reach for one after a mismatch they cannot explain. Every experiment in the quarter concluded with a lift. The quarterly metric is flat. Neither number has to be wrong for both to be true: a two-week test measures two weeks, users adjust to a change, novelty fades, and overlapping wins do not add up the way a spreadsheet of lifts says they do.

This post covers what a holdout experiment is, the four cases that justify one, the traffic math that rules it out for most small products, and how to run one as a file in your repo.

Key Takeaways

  • A holdout experiment is a control group kept on the old experience after a change ships, held for weeks or months, to measure an effect a two-week A/B test cannot see.
  • Run one for cumulative impact across a quarter, for metrics that move slowly (retention, churn, support volume), and to check whether a week-one win decays.
  • A 5/95 holdout needs roughly five times the total traffic of a 50/50 test to detect the same effect, because the small arm dominates the error.
  • The cost is a code path you keep alive on purpose, which is the stale-flag failure mode by choice. Give the file an owner, an end date, and a decision.
  • In dif a holdout is the same Markdown file as an A/B test with the weights skewed: control at 5, the shipped variant at 95, held instead of ramped.

What a holdout experiment is

A holdout experiment withholds a shipped change from a defined slice of users so that slice stays comparable to the world before the change. Everyone else gets the new thing. The difference between the two groups, measured over a long window, is the incremental effect of what you shipped.

It comes in two shapes. A per-change holdout keeps 5% of users off one feature for a quarter, to measure that feature. A global holdout keeps a slice of users off everything a team ships in a period, to measure the program. The first answers “did the redesign work,” and the second answers “did the last three months of work move the number.”

In dif, both are the same Markdown file as any other experiment, with the weights skewed and left alone:

---
id: checkout-redesign-holdout
status: active
owner: ada@acme.dev
surface: checkout
hypothesis: >
  The checkout redesign holds its week-one lift on completed_checkout
  over a full quarter, rather than decaying as novelty wears off.
variants:
  - id: control
    weight: 5
    summary: Pre-redesign checkout, held through 2026-12-15
  - id: redesign
    weight: 95
metrics:
  primary: completed_checkout
  guardrails: [refund_rate, support_ticket_rate]
exclusion_group: checkout-holdout
created: 2026-09-15
---

Same file. Different weights. The only structural difference between this and a 50/50 A/B test in code is the split and the fact that nobody touches it for three months.

Holdout vs A/B test vs staged rollout

The three look alike in the code and differ in intent:

  • An A/B test splits traffic to answer a hypothesis, then concludes. Two to four weeks, then the file is archived and the winner ramps.
  • A staged rollout moves one number up over days: 5, then 25, then 100. The goal is reaching 100 without an incident.
  • A holdout experiment is a rollout stopped short on purpose. It sits at 95 and stays there, because the 5% that never advances is the measurement.

That shared shape is why a holdout needs no new concept in a git-native tool. In dif, experiments are files, and turning a rollout into a holdout means not merging the last weight bump, plus writing down why.

The bucketing is what makes it hold. dif assigns each user with the first four bytes of SHA-256(salt || user_id) mod 10,000, recomputed on every call and stored nowhere. A user in the holdout stays in the holdout across sessions and devices for as long as the weights stay put, which is the property a three-month measurement depends on.

When to run a holdout experiment

Four cases earn the cost.

1. The quarter did not add up. A quarter of experiments all concluded with a win, and the headline metric did not move. A global holdout gives you one number for the period instead of a stack of individually optimistic ones.

2. The metric moves slower than a test window. Retention, churn, renewal rate, support ticket volume, and revenue per user take longer to respond than the two weeks a typical experiment runs. Google’s search ads team studied this directly, using A/A post-periods to separate what users learned from the immediate effect, and the long view led them to cut mobile ad load by about half (Experiment design and modeling for long-term studies in ads).

3. You suspect novelty. A redesign that wins in week one may be measuring attention, not value. A holdout tells you whether the lift is still there in week 12. The same logic runs in reverse for primacy: a change that looks bad while regular users relearn the interface can recover, and only a long control shows it.

4. The change is continuous. A model you retrain every week, or a ranking rule that keeps adjusting itself. There is no single release to A/B test, so the holdout is the only clean counterfactual available.

A holdout is the number a PM brings to the quarterly review. For the engineer it is a file nobody touches for three months, with dif validate running against it in CI the whole time so it cannot break quietly.

When a holdout is the wrong call

The first constraint is traffic, and it is unforgiving. The error on a difference between two groups scales with 1/n_control + 1/n_treatment, so a 5/95 split is dominated by the 5%. At the same total traffic, that split needs roughly five times as many users as a 50/50 test to detect the same size effect.

If your normal A/B tests already take three weeks to reach a decision, the 5% version of the same question needs a quarter of traffic behind it. Run the numbers before you commit, using something like Evan Miller’s sample size calculator, and widen the holdout to 10% or 20% when the math says so.

Three other cases rule it out:

  • Changes everyone must get. You do not hold 5% of users back from a security fix, a compliance change, or a pricing correction to see what happens. Those ship to everyone.
  • Small cosmetic changes. A holdout costs a carried code path for months. Spend that on decisions worth defending later, not copy tweaks.
  • Anything where the stale branch is dangerous. If the old path touches payments or auth and will drift out of test coverage, the holdout is a liability that grows every sprint.

A holdout also costs its own users something. If the change was good, 5% of people spend a quarter on the worse version of your product so you can find out by how much.

The cost is a code path you keep alive

A holdout experiment asks you to do the exact thing that produces the flag graveyard. The old branch stays in the codebase for months, behind a condition nobody is thinking about. That is how a flag like new-checkout-v2 ends up at 100% for three years with a dead branch behind it and nobody willing to delete either one.

The difference has to be bookkeeping, and the file is where it goes. The holdout has an owner email the validator checks, an end date written into the variant summary and the brief, a status that puts it in ls dif/experiments/active/, and a ## Decision section that stays empty until someone fills it. Because the flag is a file in the repo rather than a row in a dashboard, the quarterly question “why is this still here” is answered by git log, not by asking around. That is the same argument as feature flags in git, with a longer clock on it.

State the other side of it too: in dif, closing a holdout is a merge and a deploy, not a toggle. You edit 5 to 0, push the weight the rest of the way, and ship. If your build takes ten minutes, your fastest close is ten minutes. For a holdout that has run for 90 days, ten minutes is not the constraint. For a kill switch on a payment path, it would be, and a dashboard-based tool is the right call there.

Running a holdout in the repo

Draft the file from the surface it belongs to, so prior learnings come with it:

dif new checkout-redesign-holdout --surface checkout

→ reading dif/surfaces/checkout.md
  found 3 prior learnings
→ drafted dif/experiments/active/checkout-redesign-holdout.md
  status: draft, owner: ada@acme.dev

The draft starts at 50/50. Edit the weights to 5 and 95, write the hypothesis as a long-run claim rather than a launch claim, and set the guardrails you expect the redesign to threaten. Then check it and compile:

$ dif validate
✓ all checks passed

$ dif build
✓ validated 1 active experiment
✓ client    → dif/generated/client.ts
✓ audiences → dif/generated/audiences.ts
✓ context   → dif/context.json

Two checks matter here. Weights have to sum to 100, so a fat-fingered holdout fails with E005 in CI instead of in production. And two active experiments on the same surface must share an exclusion_group or have provably disjoint audiences, or the build fails with E007. That second one is the real hazard of a long-running holdout: a teammate ships a checkout test in week six that overlaps your control group and quietly poisons the comparison. The collision fails their pull request instead.

A program-wide holdout spans surfaces, and dif’s exclusion check is scoped to one surface, so build that one out of an audience attribute rather than a group key. Declare holdout_cohort in dif/config.yaml, give it a resolver at dif/audiences/holdout_cohort.ts, and exclude it in every other experiment’s audience block:

audience:
  exclude:
    - holdout_cohort: true

Verify an actual user before you trust the numbers:

$ dif qa --user u_8131
trace u_8131:
  • checkout-redesign-holdout → control (bucket 214)

At the end of the window, dif conclude dates the decision, moves the file to dif/experiments/concluded/, and prepends one line to the surface log, so the next person to test checkout reads what the holdout found before they draft anything.

FAQ

What is a holdout in experimentation? A group of users deliberately kept on the old experience after a change ships, so they stay a valid control for weeks or months. The difference between the holdout group and everyone else is the long-run effect of the change.

What is the difference between a holdout and a control group? A control group exists for the length of one experiment. A holdout is a control that keeps going after the experiment concludes and the winner ships, which is what lets it measure decay, learned behavior, and cumulative program impact.

How big should a holdout group be? Usually 5% to 10%, and the floor is set by traffic, not preference. Because the error is dominated by the smaller arm, a 5/95 split needs about five times the total users of a 50/50 test for the same detectable effect. Low-traffic products should widen the holdout or skip it.

How long should a holdout experiment run? Long enough for the slow metric to respond, which is typically one full quarter, and never shorter than the natural cycle of the behavior you are measuring. Decide the end date when you open the file, and write it down, or the holdout becomes permanent by accident.

Is a holdout the same as a holdback? Yes, the terms are used interchangeably. Some teams say “holdback” for a per-feature hold and “global holdout” for the program-wide version, but the mechanism is identical: a slice of traffic that never advances past control.

Do you need a dedicated platform to run a holdout experiment? No. You need deterministic assignment that does not drift over months, a metric pipeline, and a record of what was held back and why. In dif that record is a Markdown file in the repo, and the analysis runs wherever your event data already lands, whether that is your own warehouse or dif Cloud.

Run one, or decide not to

A holdout experiment is worth running when nobody on the team can answer “what did all of that shipping actually do” without shrugging. Skip it when your traffic cannot fill the small arm, when the change has to reach every user, or when the old code path is one you cannot safely maintain for a quarter.

Four things make one work: a slice big enough to measure, an end date fixed before you start, guardrails agreed in advance, and a decision written down when it closes. The first three are a planning problem. The last one is the step teams skip, which is why the same question comes back next year.

If you want the holdout to be a file your team reviews in a pull request instead of a setting someone remembers configuring, install the CLI and scaffold a workspace:

npm install -g @dif.sh/cli
dif init

The first experiment guide walks through the loop end to end, and the same CLI runs a holdout, an A/B test, and a staged rollout with no new concepts to learn. For the statistical side of long-running measurement, Ron Kohavi’s experiment platform writing is the standard reference.