Get started
← Blog

Experiment decisions as code: hypothesis to conclusion in a PR

Oct 7, 2026

Most A/B tests leave two records. The code change is in git. The hypothesis, the split and the decision are in a dashboard, a doc or a Slack thread, and a year later nobody can find them.

This tutorial keeps both in the repo. One experiment goes from a hypothesis in a Markdown file to a recorded decision, and every step is a diff a reviewer can read. The tool is dif, an MIT-licensed CLI and TypeScript SDK (JavaScript and TypeScript apps only). Every command below was run with dif 0.6.3 on Node 22.21 in an empty directory, with no account and no API key. The output is copied from the terminal and trimmed where noted.

Key Takeaways

  • The experiment is one file, dif/experiments/active/hero-cta.md. The hypothesis, the 50/50 split and the metric names are frontmatter fields.
  • dif validate exits 1 on a bad file (here, E005 for weights that sum to 90), so the mistake fails CI instead of reaching users.
  • dif build compiles the file into dif/generated/client.ts, and the SDK assigns variants locally from a hash of the user id.
  • dif conclude moves the file to dif/experiments/concluded/2026-10-hero-cta.md, writes the ## Decision section and appends one line to dif/surfaces/home.md.
  • The open-source core does no analysis. It records the decision text you type. The numbers come from your warehouse, or from the paid Dif Cloud.

Set up

$ git init -q && git config user.email dev@example.com
$ npm init -y
$ npm install -D @dif.sh/cli@0.6.3
$ npx dif --version
dif 0.6.3
$ npx dif init --agents none --events custom

dif init creates dif/experiments/active/ and concluded/, and writes dif/config.yaml, dif/.gitignore, dif/surfaces/home.md, two audience resolvers and two event handlers.

--agents none skips the agent instruction files (CLAUDE.md, AGENTS.md and others) that dif init writes by default. --events custom scaffolds the handlers, dif/events/exposure.ts and dif/events/track.ts. You own both files, and events go wherever those functions send them.

PR 1: the hypothesis

$ npx dif new hero-cta --surface home
→ reading dif/surfaces/home.md
  found 0 prior learnings
→ drafted dif/experiments/active/hero-cta.md
  status: draft, owner: dev@example.com

The draft has placeholders. Fill in the frontmatter and the brief:

---
id: hero-cta
status: active
owner: dev@example.com
surface: home
hypothesis: >
  Naming the first step ("Create your first invoice") instead of the offer
  ("Start free trial") lifts signups started from the home page.
variants:
  - id: control
    weight: 50
    summary: "Start free trial"
  - id: first_invoice
    weight: 40
    summary: "Create your first invoice"
metrics:
  primary: signup_started
  guardrails:
    - signup_completed
created: 2026-10-07
---

## Brief

The hero button on the home page says "Start free trial". The variant says
"Create your first invoice". Same button, same destination, same styling.

Below the brief, the draft’s ## Rationale section holds the interview notes behind the idea.

Two things to know about this file. dif new writes status: draft, and a draft is skipped by the build with a warning and exit 0, so set status: active yourself. And the first declared variant is the fallback, so put the control first.

There is also a typo in it. The weights are 50 and 40.

$ npx dif validate
error E005 dif/experiments/active/hero-cta.md:1:1: variant weights sum to 90, expected 100
  help: Distribute the variants so the weights total 100.

✗ 1 error(s), 0 warning(s)
$ echo $?
1

Run npx dif validate as a CI step on every pull request and this fails the check. Change 40 to 50 and the same command prints ✓ all checks passed.

What the validator does not check: signup_started is a name. Nothing in the open-source core confirms that an event with that name exists in your analytics.

Build and add the call site

$ npx dif build
✓ validated 1 active experiment(s)
✓ typed client → dif/generated/client.ts
✓ audiences   → dif/generated/audiences.ts
✓ events      → dif/generated/events.ts
✓ context     → dif/context.json

The part of dif/generated/client.ts that matters:

__register({
  id: "hero-cta",
  surface: "home",
  variants: ["control", "first_invoice"] as const,
  salt: "64118180f44348050024a5e474b86412",
  weights: { "control": 50, "first_invoice": 50 },
  exclusionGroup: null,
  created: "2026-10-07",
  audience: () => true,
});

Install the runtime with npm install @dif.sh/sdk@0.6.3 (it has no dependencies). Here is a call site as a script, hero.ts, that simulates four users. In an app, userId returns the signed-in user and the handlers forward to your analytics tool.

import "./dif/generated/client.ts";
import { dif } from "@dif.sh/sdk";

for (const id of ["u_1", "u_2", "u_3", "u_4"]) {
  dif.init({
    userId: () => id,
    events: {
      mode: "custom",
      exposure: (e) => console.log("  exposure:", e.experiment, e.variant, "bucket", e.bucket),
      track: (e) => console.log("  track:", e.metric, e.user_id),
    },
  });

  const label = dif("hero-cta", {
    control: () => "Start free trial",
    first_invoice: () => "Create your first invoice",
  });

  console.log(id, "->", label());
  dif.track("signup_started");
}

$ npm pkg set type=module
$ node hero.ts
  exposure: hero-cta first_invoice bucket 6112
u_1 -> Create your first invoice
  track: signup_started u_1
  exposure: hero-cta first_invoice bucket 6933
u_2 -> Create your first invoice
  track: signup_started u_2
  exposure: hero-cta control bucket 4230
u_3 -> Start free trial
  track: signup_started u_3

The fourth user, u_4, also gets control, in bucket 3851.

The SDK hashes the user id with SHA-256 into a bucket from 0 to 9999. Buckets below 5000 get control. The same user gets the same bucket on every run and on every machine.

QA before merge

dif qa runs the same assignment in the Rust CLI, so you can check a user without starting the app:

$ npx dif qa --user u_3
trace u_3:
  • hero-cta → control (bucket 4230)

Bucket 4230 matches what the SDK printed. To look at the other variant, force it:

$ npx dif qa --user u_3 --force hero-cta=first_invoice --preview-url https://staging.example.com/
trace u_3:
  • hero-cta → first_invoice (forced)
preview: https://staging.example.com/?_dif=hero-cta%3Dfirst_invoice
          open in the app to force these variants — fires no exposure; ?_dif=off clears

Paste that link into the pull request so a reviewer sees the variant in their own browser. A forced view fires no exposure, so QA traffic stays out of the data.

Ship

dif/generated/ is gitignored by dif/.gitignore. The pull request holds the experiment file and the call site, and the deploy pipeline has to run dif build before the app build. Add "prebuild": "dif build" to package.json.

Then merge and deploy. That is the only way an experiment starts in dif. The weights are compiled into the client, so there is no dashboard switch. Changing the split to 90/10 or stopping the test is another edit, another merge and another deploy. If you need to turn something off in seconds without a deploy, dif is the wrong tool for that job. (I did not deploy this scratch project. The deploy step is whatever your app already does.)

What dif does not do while the test runs

This is where the open-source core stops. It has no stats engine, no dashboard and no results view. It does not compute lift, confidence or sample size. metrics.guardrails is a list of names, and nothing in the core watches them.

What you get is one exposure event per user per page session and one event for each dif.track() call, delivered to the two handler functions. From there you have two options.

  1. Forward the events to the analytics tool or warehouse you already have, and run the comparison there. The join is exposures to metric events on user_id.
  2. Connect Dif Cloud, the hosted product. Per its page it ingests the same events and reports lift with a 95% confidence interval. It costs $50 a month per workspace and has no free tier.

PR 2: the conclusion

Say the test ran for three weeks and the variant won. The numbers below are sample text I typed for this tutorial. They are not a real result.

$ npx dif conclude hero-cta --decision "Shipped first_invoice to 100%. signup_started 4.1% vs 3.6% over 21 days, signup_completed flat. Numbers from our warehouse query, linked in the PR."
→ moved dif/experiments/concluded/2026-10-hero-cta.md
→ drafted ## Decision block
→ appended to dif/surfaces/home.md
  "Shipped first_invoice to 100%. signup_started 4.1% vs 3.6% over 21 days, signup_completed flat. Numbers from our warehouse query, linked in the PR."
$ git status --short
 D dif/experiments/active/hero-cta.md
 M dif/surfaces/home.md
?? dif/experiments/concluded/

Three changes, all in the working tree for you to commit. The file moved and gained a year-month prefix. Its frontmatter now reads status: concluded and concluded: 2026-10-07, and the hypothesis, variants and brief are untouched. The decision text sits under ## Decision at the end of the body.

dif/surfaces/home.md has a new line under ## Learnings:

- 2026-10-07 — hero-cta: Shipped first_invoice to 100%. signup_started 4.1% vs 3.6% over 21 days, signup_completed flat. Numbers from our warehouse query, linked in the PR.

dif conclude does not check those numbers against anything. The pull request is the check. A reviewer who doubts “4.1% vs 3.6%” asks for the query before approving.

The step that is easy to miss

Concluding does not ship the winner. Run the script again after dif conclude and dif build (first two of four users shown):

$ node hero.ts
u_1 -> Start free trial
  track: signup_started u_1
u_2 -> Start free trial
  track: signup_started u_2

Every user gets the first branch, control, and no exposure fires, because hero-cta is no longer registered. The losing copy is live for everyone. The validator points at the line:

$ npx dif validate
warning W001 hero.ts:14:1: orphan ref: `hero-cta` is not an active experiment
  help: Either create the experiment or remove the dif() call.

⚠ 1 warning(s)

It is a warning and the exit code is 0, so CI will not stop you. Put the code change in the same pull request as the conclusion: delete the dif("hero-cta", ...) call and hard-code “Create your first invoice”. That scan reads .ts, .tsx, .js and .jsx files only.

What the next experiment inherits

Run npx dif new hero-subhead --surface home and the output now says found 1 prior learning. The new draft opens with the hero-cta line in a comment under ## Brief, labelled “Recent learnings on home”. The next person to test this surface reads the last decision first.

FAQ

Does dif tell me which variant won? No. The open-source CLI and SDK assign variants and deliver events. Analysis happens in your warehouse or analytics tool, or in Dif Cloud, which is paid.

Can I stop a test without a deploy? No. Weights are compiled by dif build. Stopping a test is an edit, a merge and a deploy.

Getting started

Run npm install -D @dif.sh/cli, then npx dif init. The CLI reference covers all eight commands, and the source is at github.com/dif-sh/dif.