How to write an A/B test hypothesis worth reading

A good A/B test hypothesis names the change, the metric it should move, and the reason, in one sentence: “Changing X will move metric Y because Z.” Anything shorter is a guess; anything longer is a design doc.

Most hypotheses fail before the test runs, because they never clear that bar. “Test a new checkout button” names a change and nothing else. “We believe users will convert better” names a metric and nothing else. Neither can be proven wrong, so neither has earned the traffic it is about to spend.

This post is about the three-part format an A/B test hypothesis needs, where it actually lives in a real experiment file, three worked examples, and the mistakes that sink most hypotheses before a single user sees the test. It sits inside the lifecycle of running an A/B test as a developer, at the one step most guides skip.

Key Takeaways

• A hypothesis needs three parts in one sentence: the change, the metric it should move, and the reason, for example “adding a shipping-cost badge lifts add-to-cart rate because it removes a surprise at checkout.”
• A hypothesis that cannot be proven wrong is not a hypothesis. State a direction you are willing to be wrong about.
• In dif, the hypothesis is a prose field in the experiment’s frontmatter, reviewed in the same pull request as the code it gates.
• dif new pre-loads the surface’s last three concluded learnings into the draft, so the next hypothesis starts from what already failed, not a blank page.
• The most common failure is not a bad idea. It is an untestable one: no metric, no direction, or a sample size the traffic can’t reach in a reasonable window.

What makes an A/B test hypothesis good

An A/B test hypothesis needs three parts, in this order: the change you are making, the metric you expect it to move, and the reason you expect that movement. Drop any one and the test stops being an experiment and starts being a coin flip with a ticket number.

“We’re testing a new CTA” has a change and no metric. “We think conversion will go up” has a metric and no change. “A benefit-led CTA lifts completed checkout because it answers the hesitation a generic ‘Buy now’ leaves open” has all three, and it tells you exactly what result would prove it wrong.

The reason clause is the one teams skip, and it is the one that matters most. Without it, a win and a loss both teach you nothing beyond “that specific button did a thing.” With it, a loss tells you the hesitation was not what you thought, which is itself worth knowing.

The null hypothesis you’re testing against

Every A/B test hypothesis has a quiet twin: the null hypothesis, the claim that your change does nothing and any difference you see is noise. You are not trying to prove your hypothesis. You are trying to gather enough evidence to reject the null one.

That framing matters because it forces falsifiability. If no result, win or lose, could reject the null hypothesis, you have not written a hypothesis, you have written an opinion. Evan Miller’s guide to running an A/B test makes the related point about stopping early: a test that wanders in and out of “significant” while it runs will eventually look like a win if you check often enough and stop the moment it does. Decide the sample size before you start, or the null hypothesis never gets a fair trial.

Where the hypothesis lives: prose in the experiment file

Most hypotheses live in a ticket, a doc, or a note field in a dashboard nobody reopens once the test ships. In dif, the hypothesis is prose in the experiment’s own frontmatter, next to the variants and the metric it is meant to move, the same way experiments as files covers the whole lifecycle.

---
id: checkout-shipping-badge
status: active
owner: sam@acme.com
surface: checkout
hypothesis: >
  Showing "free shipping over $50" on the cart page lifts
  add-to-cart rate, because shipping cost is the most common
  checkout abandonment reason and surfacing it earlier removes
  the surprise.
variants:
  - id: control
    weight: 50
  - id: shipping_badge
    weight: 50
metrics:
  primary: add_to_cart_rate
  guardrails:
    - checkout_abandon_rate
created: 2026-09-24
---

A PM writes the hypothesis clause; an engineer reviews the weights and the metric name against the actual event. Both read the same file, in the same pull request, before the test ships. That is the .md format doing the job a dashboard’s free-text box never does: making the hypothesis reviewable, not just recorded.

That review has a cost. Writing a reason clause that could be proven wrong takes longer than typing a one-line test name into a dashboard, and a reviewer can send the file back before the test ever ships. The cost buys a result someone can act on instead of a data point nobody revisits.

Three A/B test hypothesis examples, broken down

• Pricing page. “Adding a badge that shows the annual discount in dollars, not a percentage, lifts annual-plan signups, because a percentage makes users do the math and a dollar amount does not.” Change: the badge’s copy. Metric: annual-plan signups. Reason: removing a mental calculation.
• Onboarding. “Skipping the product tour for users who have signed in before raises day-one activation, because a tour they have already seen adds friction instead of context.” Change: tour visibility, gated on prior sign-in. Metric: day-one activation. Reason: friction without new information.
• Search results. “Sorting results by recency instead of relevance lowers zero-click search sessions, because most queries on this surface are about something that happened today, and a static relevance match misses that.” Change: default sort order. Metric: zero-click sessions. Reason: query intent skews time-sensitive.

Each one names a testable direction. If annual-plan signups do not move, the dollar-amount theory is wrong, not just “the test didn’t work.” That is the difference a real reason clause buys you.

Hypotheses that don’t survive contact with data

Four patterns account for most of the hypotheses that fail before the results are even worth reading:

• No direction. “We’ll see what happens to conversion” is not a hypothesis. State which way you expect the metric to move, or you have no way to call the test a loss.
• Two changes, one hypothesis. Changing the CTA copy and the button color in the same variant means a win cannot tell you which change caused it. One change per hypothesis, every time.
• A metric too rare to move in time. A metric that only fires a few dozen times a week will not reach significance before the business cycle changes underneath the test. Check the guardrail and significance guidance before committing traffic to it.
• A reason nobody could disprove. “Users will like it more” is not falsifiable. “Users complete checkout at a higher rate because the new copy resolves a specific hesitation” is.

Write it with yesterday’s learning, not a blank page

The hardest part of writing a hypothesis is knowing whether someone already tried it, not writing the sentence itself. A surface with six months of concluded tests has already disproven some of the ideas a new hire is about to propose again.

dif new reads a surface’s last three concluded learnings and drops them into the draft before you write a word:

$ dif new checkout-shipping-badge --surface checkout
→ reading dif/surfaces/checkout.md
  found 3 prior learnings
→ drafted dif/experiments/active/checkout-shipping-badge.md
  status: draft, owner: sam@acme.com

The draft opens with those lines in a comment under ## Brief:

Recent learnings on checkout:
- 2026-06-02 — checkout-cta-copy: Benefit-led CTA copy lifted completed_checkout 2.1%. Shipped.
- 2026-04-14 — checkout-progress-bar: Removing the progress bar had no effect on completion. Reverted.
- 2026-02-28 — checkout-second-provider: A second payment provider option raised refund_rate. Reverted.

Reading that the progress bar test already failed saves the next hypothesis from repeating it, and it points the reason clause somewhere more specific than a guess. dif conclude is what writes each learning back to the surface in the first place, so the file that pre-loads your draft is the same file your last test updated.

FAQ

What is an A/B test hypothesis? A one-sentence statement naming the change you are making, the metric you expect it to move, and the reason you expect that movement. It is what the test is meant to prove or disprove, not a description of the feature.

What’s the difference between a hypothesis and a null hypothesis? The hypothesis is the direction you expect; the null hypothesis is the default claim that your change does nothing and any difference is noise. A test is designed to gather enough evidence to reject the null hypothesis, not to confirm your own.

How specific should an A/B test hypothesis be? Specific enough that one clear result, in either direction, would prove it wrong. “Conversion will improve” is too vague. “Add-to-cart rate rises because the shipping cost surprise is removed” names a metric and a mechanism you can check.

Can you test more than one hypothesis at once? Not in the same variant. Bundling two changes into one test means a win cannot tell you which change caused it. Run them as separate experiments, or as sequential tests on the same surface.

Where should the hypothesis live, the ticket or the code? As close to the code as you can get it. A hypothesis in a ticket gets lost when the ticket closes. A hypothesis in the experiment file, reviewed in the same pull request as the code it gates, stays attached to the result.

What happens if the hypothesis is wrong? A well-written hypothesis that loses still tells you the reason clause was wrong, which is specific enough to inform the next one. That is the point of writing the reason down in the first place.

Writing an A/B test hypothesis that holds up

A good A/B test hypothesis names the change, the metric, and the reason, in one sentence you could be proven wrong about. Skip the reason clause and a loss teaches you nothing. Bundle two changes into it and a win teaches you nothing either.

Write it where the code lives, review it in the same pull request as the variants it describes, and read what the surface already learned before you draft the next one. dif new does that last part automatically, pulling the last three concluded learnings into the file before you write a word.

Install the CLI and try it on a real surface: npm install -g @dif.sh/cli, then dif init. The first experiment guide walks through writing a hypothesis, holding the split, and concluding with a decision, end to end.