Get started
← Blog

Agent-driven experiments: your coding agent writes both variants

Sep 8, 2026

An agent-driven experiment is an A/B test where a coding agent like Claude Code or Cursor writes both variants behind a dif flag, dif runs the split, and the result comes back into the agent’s context on the next session. The developer supplies the hypothesis, the primary metric, and the guardrail. The agent writes variant A, variant B, and the Markdown file that gates the split. dif routes traffic by deterministic bucketing, records exposures, and after the test concludes writes the decision and a one-line learning back to the surface.

Agent-driven experiments work because a dif experiment is a file, not a dashboard row. Everything the agent needs, the config, the two code paths, the results, is text in the repo it already reads.

Key Takeaways

  • Agent-driven experiments mean the coding agent authors variant A, variant B, and the Markdown experiment file. dif runs the deterministic bucketing and records the exposures.
  • A dif experiment is one .md file with a hypothesis, a primary metric, weights, and a guardrail. Different frontmatter, same shape as a flag.
  • Results come back into the agent’s context because dif build writes context.json and dif conclude writes a one-line learning back to the surface. Both are files the agent reads on session start.
  • The agent speeds up variant authoring and analysis, not sample size. A 50/50 test on 500 daily users still needs the days a t-test needs.
  • The human writes the hypothesis, names the primary metric, and runs dif conclude. The agent proposes, the human decides.

What agent-driven experiments change, and what they do not

An agent is good at three parts of an experiment. It writes the variant code because it reads the render site and the surrounding types. It writes the Markdown experiment file because the schema is a fixed shape it can fill in. It drafts the readback of a concluded learning because that log is text.

An agent is bad at three parts. It cannot generate traffic. It cannot decide what matters, because the primary metric is a judgment about the business, not a value the code can compute. It cannot conclude the test, because ending an experiment changes what customers see, and that call belongs to a person.

The workable split: the human states the hypothesis, names the primary metric, and runs dif conclude. The agent writes the variants, the flag file, and the next session’s readback. Everything in between is bucketing math running the same way it always has.

The end-to-end loop for an agent-driven experiment

The loop has six steps. Five belong to the agent. One belongs to the human.

  1. The human states the hypothesis. One sentence, in chat or in the PR. Include the primary metric and one guardrail. “Benefit-led CTA lifts completed_checkout without moving cart_abandonment more than 0.5%.”
  2. The agent scaffolds the experiment. It runs dif new experiment checkout-cta-copy, opens the drafted file, and fills in variants, weights, hypothesis, primary metric, and guardrail.
  3. The agent writes both variants. It edits the render site to gate on dif("checkout-cta-copy") and adds the second copy path. Both branches compile.
  4. The agent runs dif validate. Weights total 100, the owner is a real email, the source has a matching call site, and no other active experiment on the checkout surface collides. A broken flag fails the PR check the same way a broken build does.
  5. The PR opens. The reviewer reads the hypothesis, the render-site diff, and the flag file. Merge triggers dif build, and context.json regenerates with the new experiment listed.
  6. The human runs dif conclude. After the test hits its sample-size target, the human picks the winner and writes the one-line learning. dif conclude archives the file, appends the learning to the surface log, and regenerates context.json.

The agent never picks the winner. It reads the winner on the next session. This is the shape feature flags for AI agents applies to experiments.

A real hypothesis-to-conclusion trace

The team runs an e-commerce site. Checkout completion sits at 8.2%. The PM wants to test benefit-led CTA copy against the control.

The developer opens Claude Code and pastes:

New experiment on checkout: benefit-led CTA copy vs control.
Primary metric completed_checkout. Guardrail add_to_cart_abandonment.
Author the file, gate the render site, open the PR.

The agent reads AGENTS.md, sees the managed dif block, and knows the schema. It reads context.json, sees no active experiment on the checkout surface, and confirms no exclusion-group conflict. It calls dif new experiment checkout-cta-copy, edits the drafted file, opens the checkout render site, and gates the copy on a dif("checkout-cta-copy") call. It runs dif validate. Two errors first: the owner is blank, and one variant has weight 60 while the other has 40. It fixes both and validates clean. It opens the PR.

Two weeks later, the primary metric hits its sample-size target. dif Cloud writes an aggregated results block back into the PR:

checkout-cta-copy: 14 days, n=42,318
  control     8.20% completed_checkout
  benefit_led 8.37% completed_checkout (+2.1%, p=0.014)
  guardrail add_to_cart_abandonment  +0.3% (noise band)

The PM reads that block and runs dif conclude with the winner and a one-line learning. The next agent session reads that learning on the checkout surface.

The artifacts, in real content

Three files carry the experiment. Here they are, filled in.

The experiment file at dif/experiments/active/checkout-cta-copy.md:

---
id: checkout-cta-copy
type: experiment
status: active
surface: checkout
owner: chris@example.com
created: 2026-09-08
hypothesis: >
  A benefit-led CTA ("Get it today") lifts completed_checkout
  more than the control ("Place order").
primary_metric: completed_checkout
guardrail_metrics:
  - add_to_cart_abandonment
audience: default
exclusion_group: checkout-copy
variants:
  - id: control
    weight: 50
    copy: "Place order"
  - id: benefit_led
    weight: 50
    copy: "Get it today"
---

Rationale: a prior test on the pricing surface (surfaces/pricing.md,
2026-04-11) showed benefit-led copy lifted click-through 3.4%.
Testing whether the same pattern moves completed_checkout.

The coding-agent transcript in Claude Code, abbreviated:

> Author checkout-cta-copy experiment: benefit-led CTA vs control.
  Primary metric completed_checkout, guardrail add_to_cart_abandonment.

Reading AGENTS.md and dif/context.json.
No active experiment on checkout. exclusion_group `checkout-copy` unused.

Running: dif new experiment checkout-cta-copy
  Created: dif/experiments/active/checkout-cta-copy.md

Editing src/routes/checkout/+page.svelte:
  const ctaCopy = dif('checkout-cta-copy') === 'benefit_led'
    ? 'Get it today'
    : 'Place order';

Running: dif validate
  E101 owner field is empty          -> fixed: owner=chris@example.com
  E403 weights differ from prompt    -> fixed: control 50, benefit_led 50

Running: dif validate
  Ok. 1 experiment, 0 errors, 0 warnings.

Opening PR #482: Add checkout-cta-copy experiment.

The agent used dif’s installed skills to know which verb runs the scaffold and how the frontmatter is shaped. Letting the agent author the flag covers the flag version of the same move.

The results readback the agent reads on the next session, from dif/context.json:

{
  "surfaces": {
    "checkout": {
      "last_learning": "2026-09-22 benefit_led lifted completed_checkout 2.1% (n=42318, p=0.014). Shipped.",
      "landmines": [
        "guardrail add_to_cart_abandonment moved +0.3%, inside noise band"
      ]
    }
  }
}

dif conclude wrote that line. On the next agent session, the first thing Claude Code reads is that learning. Ask it to test benefit-led CTA copy again and it says the test already concluded and cites the file. Full mechanics in what context.json is.

Failure modes to gate on

Four failure modes recur when the agent runs the authoring. Catch each in review or in CI.

Peeking. The agent reads intermediate results and wants to call it early. Do not let it. A primary metric up 2.1% at day three and 0.3% at day fourteen is the same test, and the day-three call would have been wrong. Do not surface an early p-value the agent will pattern-match on. The human runs dif conclude, not the agent.

No guardrail metric. The agent optimizes exactly what you name. Ask for completed_checkout and it will happily ship a variant that doubled cart abandonment on the way to it. Every experiment file needs at least one guardrail_metrics entry, and it belongs in the PR template so review catches its absence. A/B testing for developers covers guardrails at length.

No exclusion group. Two concurrent experiments on the same surface contaminate each other’s metrics. If both files declare the same exclusion_group, dif guarantees each user is in at most one. Without it, an agent that authored two checkout experiments in the same PR ships a mashup to some users. dif validate catches the collision (E007) if either file is missing the group.

Missing hypothesis. An agent will happily generate a plausible-sounding hypothesis when the file demands one. That is worse than no hypothesis, because “plausible” survives review. The hypothesis line is the human’s. Paste it into the prompt or write it by hand before the agent scaffolds the file.

FAQ

Can Claude Code actually run an A/B test end to end? It runs every part except sample size and the concluding decision. It authors the experiment file, edits both variants, runs dif validate, and opens the PR. dif does the bucketing and records exposures. When the sample-size target hits, a human runs dif conclude.

What does the developer supply, and what does the agent produce? The developer supplies the hypothesis, the primary metric, and one guardrail. The agent produces the Markdown experiment file, the render-site edits for both variants, and the PR. On the next session it produces the readback of the concluded learning.

How do results come back into the agent’s context? dif conclude writes the decision and a one-line learning to the surface file. dif build includes that learning in context.json, and the agent reads context.json on session start.

Does the agent speed up my sample-size math? No. It speeds up variant authoring and analysis. A 50/50 test on 500 daily users still needs the days a t-test needs.

Can an agent conclude an experiment on its own? No. dif conclude is a human verb, because picking a winner is a business decision. The agent can draft the one-line learning and propose it, but the human runs the command.

Does this work in Cursor as well as Claude Code? Yes. dif init writes managed blocks into CLAUDE.md, AGENTS.md, and .cursorrules, and context.json is a plain file. Any coding agent that reads Markdown from your repo sees the same state.

Getting started

Agent-driven experiments are the ordinary consequence of an experiment being a file. If the agent can read your code, it can read your experiments. It writes both variants and the Markdown file that gates them. What it cannot do is generate traffic, decide what matters, or call the winner.

Install the CLI and scaffold:

npm install -g @dif.sh/cli
dif init

dif init writes the dif/ directory, the managed blocks in CLAUDE.md, AGENTS.md, and .cursorrules, and the generated client. The next dif build writes the context.json your agent reads. Then ask it to author the first experiment.