Guardrail metrics: the numbers that stop a test

A guardrail metric is a number you track during an A/B test to catch harm the primary metric cannot see, like refund rate climbing while checkout conversion goes up. It is not the metric you are trying to move. It is the one that proves the win is real, not borrowed from somewhere else.

Most teams pick one metric and ship the moment it clears significance. Conversion is up 2%, so the test wins. Then support tickets double, because the faster checkout buried the return policy. A guardrail metric is the line in the experiment file that catches that before the winner ramps to 100%.

This post covers what a guardrail metric is, how it differs from the primary metric, how dif encodes one in the experiment file, and what to do when a guardrail trips.

Key Takeaways

• A guardrail metric is a secondary number (refund rate, latency, support volume, churn) that has to hold steady while the primary metric is judged for significance.
• In dif, guardrails are a list under metrics.guardrails in the experiment file, next to the single metrics.primary.
• A guardrail doesn’t need its own significance test. It needs a threshold agreed on before the test starts, the same as the primary metric does.
• dif Cloud tracks every guardrail alongside the primary metric and writes the result back to your PR, so a regression shows up in review, not after the ramp.
• Most blown guardrails are a process failure, not a surprise: the guardrail existed in the file, nobody checked it before concluding.

What a guardrail metric is

A guardrail metric answers a different question than your primary metric. The primary metric asks “did this work?” The guardrail asks “did it work at someone else’s expense?”

Take a checkout-copy test with completed_checkout as the primary metric. A benefit-led CTA lifts completions 2%, the test is significant, and by the primary metric alone it’s a clean win. But if refund_rate climbed over the same window, the lift was partly shoppers who bought on hype and asked for their money back a week later. The guardrail catches that before anyone ramps the CTA to everyone.

A guardrail isn’t a second hypothesis. Nobody is trying to move it. It sits there to confirm the thing you did move didn’t cost you somewhere you weren’t looking.

Primary metric vs guardrail metric

The two roles split cleanly:

• Primary metric. The one number the hypothesis is tested against. The test runs until this reaches significance, and the decision rides on it.
• Guardrail metric. A number that must not regress past an agreed threshold. It isn’t tested for significance, it’s checked for harm.

A metric’s role is assigned per test, not fixed by what it is. refund_rate is a guardrail on a checkout-copy test and the primary metric on a test aimed directly at reducing refunds. Nothing about the number changes. What changes is which question it’s answering this time.

How dif encodes a guardrail in the experiment file

In dif, guardrails are a field in the same experiment file as the hypothesis and the primary metric, not a separate dashboard panel someone has to remember to check.

---
id: pricing-annual-discount
status: active
owner: priya@acme.com
surface: pricing
hypothesis: >
  A 20% annual discount at checkout lifts annual-plan upgrades
  over the monthly default.
variants:
  - id: control
    weight: 50
  - id: annual_discount
    weight: 50
metrics:
  primary: annual_plan_upgrades
  guardrails:
    - support_tickets_opened
    - monthly_churn
created: 2026-09-30
---

metrics.primary is the single number this test is judged on. metrics.guardrails is a list, so a test can watch more than one kind of harm: a spike in support_tickets_opened (the discount copy confused people) and a rise in monthly_churn (people took the annual plan, then downgraded to dodge the lock-in later). Both live next to the hypothesis in the experiment file format, so “what are we watching for harm” is answered by opening one file.

Picking the right guardrails

A guardrail earns its place by naming a specific, plausible way this exact change could hurt you. A short list beats a long one:

• Revenue and refunds. A copy or pricing change that could push people to buy on hype and ask for their money back.
• Support load. Anything that changes a flow people already call support about.
• Latency. A heavier component or an extra request that could slow the page.
• Churn or unsubscribes. A change that could win attention now and cost retention later.

Pick guardrails this specific change could plausibly move, not every metric the company tracks. A button-color test on a page with no copy change probably doesn’t need a support-ticket guardrail. A pricing page does.

What happens when a guardrail trips

Nothing stops automatically. dif doesn’t kill a test the moment a guardrail crosses its threshold, and that’s a real tradeoff worth stating plainly: the guardrail is a number in a file, not a circuit breaker wired into your deploy.

What it buys you is visibility at the point someone has to look. dif conclude asks for the decision and records it in the same file as the hypothesis. If monthly_churn moved against the test, the recorded decision is “rejected” or “hold,” with the reason next to the number that caused it, not in a Slack thread that loses to next sprint’s standup. A win that costs you a guardrail is not a win, and the file makes that easy to see before the ramp.

Connect dif Cloud and it tracks every guardrail alongside the primary metric automatically, writing the result back to your PR, so the regression is visible in code review instead of waiting for someone to open a dashboard.

Common guardrail mistakes

Too many guardrails. Every guardrail you add widens the odds that something moves by noise alone, the same multiple-comparisons problem that sinks primary metrics. Evan Miller’s how not to run an A/B test covers why checking too many numbers too often inflates false positives. Three guardrails, chosen for plausible harm, beats ten chosen out of caution.

No guardrail at all. A test with only a primary metric ships a regression the moment the change is the kind that could plausibly cost something elsewhere: pricing, copy, or anything upstream of support.

Treating the guardrail like a second primary metric. A guardrail doesn’t need to clear statistical significance to count. If it moved in the wrong direction by an amount you’d act on outside a test, that’s enough to hold the decision, the way trustworthy experimentation practice recommends treating a guardrail breach as a stop condition regardless of the primary result.

FAQ

What is a guardrail metric in A/B testing? A guardrail metric is a number you track during a test to catch harm the primary metric can’t see, like refund rate, latency, or support volume. The test isn’t judged on it, but a guardrail moving the wrong way is a reason to hold or reject the result even if the primary metric wins.

What is the difference between a primary metric and a guardrail metric? The primary metric is the number the hypothesis is tested against, and the test runs until it reaches significance there. A guardrail metric is watched for regression, not judged for significance. It exists to catch a win that quietly costs you somewhere else.

How many guardrail metrics should an experiment have? Usually one to three. Every guardrail widens the chance something moves by noise alone, so pick the ones this specific change could plausibly hurt, not every metric the company tracks.

What happens if a guardrail metric regresses during a test? Nothing stops automatically. Someone has to look at the number before concluding, decide if the regression is real, and record “rejected” or “hold” as the decision instead of ramping a result that costs more than it earns.

Can a guardrail metric also be a primary metric in a different test? Yes. A metric is a guardrail or a primary metric by role in one experiment, not by nature. Refund rate is a guardrail on a checkout-copy test and could be the primary metric on a test aimed directly at returns.

Do you need a guardrail metric for every test? Most tests benefit from at least one. A cosmetic change with no plausible downstream cost, like a button color with no copy change, may not need one. Anything touching price, copy, or a flow with a cost downstream should have one.

Getting started

A guardrail metric is the check that keeps a winning test from being a quiet loss somewhere else. Pick one to three per test, tied to what this specific change could plausibly break, not a list copied onto every experiment. In dif, the guardrail lives in the same file as the hypothesis, under metrics.guardrails, so “did we check it?” is answered by opening the file.

The discipline fails the same way most experiment hygiene fails: the guardrail only works if someone looks at it before concluding, since dif doesn’t stop a test on its own. Agree on the threshold before the test starts, the same as the primary metric, and conclude with the guardrail in view instead of in a tab nobody reopened.

If you’re already holding tests at a split, add a guardrail to the next one:

npm install -g @dif.sh/cli
dif init

The CLI reference covers dif conclude in full.

If you haven’t run a developer A/B test before, A/B testing for developers covers the rest of the lifecycle.