An A/B testing example: the file, not just the theory
An A/B testing example should show the whole mechanism: the hypothesis, the file, the code that reads it, and the decision written back when the test ends. Most A/B testing content stops at the theory (control versus treatment, statistical significance) and never shows the artifact.
Here is one, start to finish. A pricing page test, as a real dif experiment file, through validation, a reviewer’s preview, and the concluded decision.
Key Takeaways
- A complete A/B testing example is one file: frontmatter (
owner,surface,variants,metrics) plus a one-paragraph brief, not a slide deck.- The test lives at the call site.
dif("pricing-annual-default")branches the render;dif.track("annual_signup")records the outcome.dif validatecatches a bad file before it ships: weights that don’t total 100, a missing owner, an orphan call site.dif qapreviews a forced variant with no production traffic touched, so a reviewer sees the treatment before 1% of real users do.dif concludewrites the decision and a one-line learning back to the surface, so the next pricing test starts from what this one found.
The file: a complete A/B testing example
The hypothesis: defaulting the pricing page’s billing toggle to annual, instead of monthly, lifts annual plan signups without scaring off trial starts.
---
id: pricing-annual-default
status: active
owner: dana@acme.com
surface: pricing
hypothesis: >
Defaulting the billing toggle to annual lifts annual-plan
signups over the monthly default, without increasing
trial abandonment.
variants:
- id: control
weight: 50
- id: annual_default
weight: 50
metrics:
primary: annual_signup
guardrails:
- trial_abandon_rate
---
## Brief
Ship at 50/50. Two weeks minimum. Conclude when annual_signup
reaches significance and trial_abandon_rate holds flat.
That is the whole test: one Markdown file, checked into git next to the pricing page it changes. dana@acme.com is the owner dif validate will check for. pricing is the surface this test’s learning gets written to when it concludes. Nothing about the hypothesis, the split, or the metric lives anywhere else.
The code: where this A/B test example lives
The file does nothing by itself. A call site reads it and branches the render; a track call records the outcome.
const toggleDefault = dif("pricing-annual-default") === "annual_default"
? "annual"
: "monthly";
// when the signup completes:
dif.track("annual_signup");
That’s the entire implementation. There’s no separate assignment service, and the page never gets rewritten client-side after it has already painted the monthly toggle. The variant is decided before render, so the page that ships is the page the user sees, not a flash of one followed by a swap to the other.
Validate and preview before a user sees it
A file with a typo is still just a file until something checks it. dif validate runs the same pass CI runs on every pull request:
$ dif validate
✓ pricing-annual-default: weights sum to 100
✓ pricing-annual-default: owner dana@acme.com is valid
✓ pricing-annual-default: call site found in src/routes/pricing/+page.svelte
0 errors, 0 warnings
If the weights summed to 90, or the call site had been renamed in the branch but not in the file, this step fails the pull request the same way a broken build does. A reviewer never has to trust that the numbers are right; the validator already checked.
Before any real user sees annual_default, a reviewer can force it and look:
$ dif qa --user u_7742 --attr plan=trial
user u_7742 → pricing-annual-default: annual_default (forced)
preview: https://acme.com/pricing?_dif=pricing-annual-default:annual_default
The preview link mints the variant in the browser and fires no exposure event, so a design review does not quietly contaminate the experiment’s own data.
The decision: how this A/B testing example ends
Say the two-week window closes with annual_signup up and significant, and trial_abandon_rate flat. dif conclude is one command:
$ dif conclude pricing-annual-default --decision ship --note \
"annual_default lifted annual_signup; trial_abandon_rate held flat."
That archives the file with the decision and the date, and writes a one-line learning back to the pricing surface. The next person who drafts a pricing test runs dif new, and that learning is already loaded into the draft. The test does not just end; it leaves a record.
What this example gets right
Four things make this file more than a snapshot of one test.
Assignment is deterministic: a hash of the user id and a salt, recomputed the same way every time, so nobody flips between the monthly and annual default across a page reload. A guardrail sits right next to the primary metric, trial_abandon_rate alongside annual_signup, so a lift on one can’t hide a loss on the other. The file has an owner, dana@acme.com, accountable for it the same way anyone is accountable for code in a pull request.
And the file doesn’t sit at active forever. It closes, and what it taught the team survives in the surface’s log instead of an old Slack thread.
Where a real A/B test example gets harder
This example is intentionally simple, and real tests usually are not. Two weeks is a minimum, not a guarantee. If the pricing page gets a few hundred visits a day, annual_signup may never reach significance in that window, and the honest move is to extend the test or admit it was underpowered, not to call it the moment the numbers look good. Evan Miller’s how not to run an A/B test and his sample size calculator are worth running before the test starts, not after.
Running two tests on the same surface at once is the other way a sample A/B test example breaks. If a second pricing test launched that same week and competed for the same visitors, an exclusion group would keep each visitor in at most one of them, so a result is never a mashup of two changes.
Run your own
The A/B testing for developers post covers the mechanism this example used: why assignment is deterministic, why server-side avoids flicker, and what guardrails are for. This post is the file that mechanism produces, from hypothesis to decision.
To build one on your own pricing page, checkout flow, or onboarding screen:
npm install -g @dif.sh/cli
dif init
dif new drafts the file, pre-loaded with the surface’s last three learnings. The CLI reference covers validate, qa, and conclude in full, and the first experiment guide walks the same lifecycle end to end. Experimentation in dif is the product page if you want the overview first.
FAQ
What does an A/B test look like in code? A conditional at the call site and a track call at the outcome. dif("test-id") returns the variant and branches the render; dif.track("metric_name") fires when the outcome happens. Everything else (the hypothesis, the split, the metric) lives in the file, not the code.
What should an example A/B test file include? An owner, the surface it belongs to, two or more variants with weights that sum to 100, a primary metric, and at least one guardrail metric. A one-paragraph brief stating the hypothesis and when to conclude keeps the file self-explanatory months later.
How do you write an A/B testing hypothesis example? State the change, the metric you expect it to move, and the direction, in one sentence: “defaulting the billing toggle to annual lifts annual-plan signups.” A hypothesis without a direction is not a test, it is a coin flip with extra steps.
Can one file be both a flag and an A/B test example? Yes. The format is identical; a flag is one you’re ramping toward 100%, an experiment is one you’re holding at a split with a hypothesis. If annual_default wins, you ramp the same file past 50/50 without touching the call site.
What happens if you skip concluding the test? The file stays active indefinitely, the decision never gets recorded, and the next person to touch that surface has no record the test ever ran. dif conclude is the step that turns a test into institutional memory instead of a stale file.
The takeaway
An A/B testing example is not a diagram of control versus treatment. It’s a file with an owner, a hypothesis, and a guardrail. A call site with one conditional and one track call. A validate pass that catches a bad weight before it ships, and a qa trace a reviewer can run without touching production. A conclude command that writes the decision down instead of letting it evaporate. Every piece in this post came from one file, pricing-annual-default, start to finish.
If you’re running a pricing, checkout, or onboarding test right now, the fastest way to see your own version of this example is to run dif init and write the file.