Get started
← Blog

Experimentation with Claude Code and Cursor

Sep 16, 2026

Experimentation with Claude Code or Cursor divides like this. The agent writes both variants and the Markdown experiment file. dif buckets each user and records the exposure. You write the hypothesis and call the winner.

The wiring that makes it work is one command. dif init merges a block into CLAUDE.md, writes three skills under .claude/skills/, writes a project rule at .cursor/rules/dif.mdc, and merges the same guidance into AGENTS.md for every other tool. After that the agent knows the schema, the six verbs, and the error codes, because all of it is text in the repo it already reads. This is what experimentation in git buys you that a dashboard cannot.

This post is about the wiring for each tool and what a working session looks like in it.

Key Takeaways

  • dif init scaffolds four agent targets at once: CLAUDE.md plus three skills under .claude/skills/, AGENTS.md, .cursor/rules/dif.mdc, and .github/copilot-instructions.md. Narrow it with dif init --agents claude,cursor.
  • Claude Code loads .claude/skills/dif-author-experiment/SKILL.md on demand, by matching its description. Cursor attaches .cursor/rules/dif.mdc when a task touches dif/**, because the rule carries globs: dif/** and alwaysApply: false.
  • dif’s guidance sits between <!-- dif:start --> and <!-- dif:end --> markers in co-owned files, so a re-run refreshes dif’s block and leaves the rest of your CLAUDE.md alone.
  • dif validate is the gate on the agent’s work: E005 (weights do not total 100), E006 (undeclared audience attribute), E007 (two same-surface experiments with no shared exclusion group). It collects every error in one pass and exits 1, so a bad file fails the PR.
  • dif conclude <id> --decision "..." is yours to run. It archives the file to dif/experiments/concluded/<YYYY-MM>-<id>.md and appends one dated line to the surface’s Learnings log, which the next dif new embeds into the draft.

What dif init writes for Claude Code and Cursor

One command scaffolds the workspace and every agent file:

$ npm install -g @dif.sh/cli
$ dif init

✓ created dif/experiments/{active,concluded}
✓ created dif/surfaces/
✓ created dif/audiences/
✓ wrote dif/config.yaml
✓ wrote dif/surfaces/home.md
✓ wrote dif/audiences/locale.ts
✓ wrote dif/audiences/device_type.ts
✓ wrote .claude/skills/dif-{author,conclude}-experiment, dif-generate-surfaces/
✓ wrote .cursor/rules/dif.mdc
✓ merged dif guidance into CLAUDE.md
✓ merged dif guidance into AGENTS.md
✓ merged dif guidance into .github/copilot-instructions.md

next steps:
  1. npm install @dif.sh/sdk
  2. dif new <id> --surface home, then edit the file and set status: active
  3. dif build

Four targets, written by default: claude, general, cursor, and copilot. Pass --agents claude,cursor if you only want two, or --agents none to skip them.

Two of those files are dif’s alone, so they get written whole: the skill directories and .cursor/rules/dif.mdc. The rest are co-owned with you. CLAUDE.md, AGENTS.md, and .github/copilot-instructions.md already have your team’s content in them, so dif appends a block delimited by <!-- dif:start --> and <!-- dif:end --> and only ever rewrites what is between those markers. Run dif init again after a CLI upgrade and your hand-written sections survive.

Experimentation with Claude Code, in one session

Claude Code reads CLAUDE.md at session start, and that block is deliberately short. The depth lives in three agent skills that load only when the work calls for them. Each is a SKILL.md with a description Claude matches against the task, which is the standard Agent Skills shape:

---
name: dif-author-experiment
description: Authoring and iterating on dif.sh experiments, drafting frontmatter,
  choosing variants and weights, declaring audiences, and validating the file. Use
  whenever the user wants to create, draft, edit, or fix a dif experiment, or
  mentions A/B test, variant, audience, hypothesis, surface, exclusion group, or
  the commands dif new, dif validate, or dif build.
---

So the prompt is the thing you would have said anyway:

> Draft an experiment on the pricing surface: default the plan toggle to
  annual. Primary metric paid_conversion, guardrail refund_rate.

Using skill: dif-author-experiment
Reading dif/context.json: 1 active experiment on home, none on pricing.
Reading dif/surfaces/pricing.md: Learnings log, 4 entries.

Running: dif new pricing-annual-default --surface pricing
  Created dif/experiments/active/pricing-annual-default.md

Editing src/routes/pricing/+page.svelte

Running: dif validate
  E005  variant weights sum to 90, expected 100
        dif/experiments/active/pricing-annual-default.md:1

Running: dif validate
  Ok. 2 experiments, 0 errors, 0 warnings.

The skill tells it to read dif/context.json and the surface’s Learnings log before drafting, which is why the file it produces cites what the team already tried. Here is that file:

---
id: pricing-annual-default
status: active
owner: ada@acme.dev
surface: pricing
hypothesis: >
  Defaulting the plan toggle to annual lifts paid_conversion on the
  pricing page by 1% to 3% over 14 days, without raising refund_rate.
variants:
  - id: control
    weight: 50
    summary: "Toggle defaults to monthly"
  - id: variant_a
    weight: 50
    summary: "Toggle defaults to annual"
metrics:
  primary: paid_conversion
  guardrails:
    - refund_rate
exclusion_group: pricing-layout
created: 2026-09-14
concluded: null
---

## Brief

Annual is the cheaper plan per month and the toggle hides it behind a
click. Testing whether the default carries the decision.

## Rationale

Prior test on this surface (pricing-tier-order, concluded 2026-08-02)
moved nothing, which suggests ordering is not the bottleneck.

## Decision

And the render site, where the branches are thunks and the SDK picks one:

import "./dif/generated/client";
import { dif } from "@dif.sh/sdk";

const planToggle = dif("pricing-annual-default", {
  control: () => "monthly",
  variant_a: () => "annual",
});

const defaultPlan = planToggle();

For a growth PM, the reviewable unit is the top half of that Markdown file: a hypothesis, a primary metric, one guardrail, and a 50/50 split. For an engineer it is the diff at the render site and a dif validate that passes. Both land in the same PR. The agent skills walkthrough covers the same loop step by step.

Cursor reads a rule file, not a skill

Cursor has no skills directory, so dif init writes a project rule instead, at .cursor/rules/dif.mdc:

---
description: Feature flags and A/B experiments in this repo are dif.sh files under dif/. Use when adding, ramping, gating, or removing a feature flag or experiment, or when editing anything under dif/.
globs: dif/**
alwaysApply: false
---

<!-- generated by dif v0.6.0; regenerate with `dif init --force` -->

# Working with dif.sh in this repo
...

Those three frontmatter fields decide when Cursor pulls the rule in. Per Cursor’s rules docs, globs attaches a rule automatically when a task touches matching files, and a description with alwaysApply: false lets the agent request it when the description fits the task. dif sets both, so editing anything under dif/ attaches the rule, and asking for “an A/B test on pricing” is usually enough to pull it in otherwise.

The practical difference is discovery, not depth. Claude Code finds the long workflow guidance on its own because the skill description matches. Cursor gets a shorter rule, and the rule points at .claude/skills/dif-author-experiment/SKILL.md. That file is ordinary Markdown in the repo, so Cursor can open and read it. The same is true of any agent reading AGENTS.md: the block names the file layout, the verbs, and the error codes, and everything deeper is one read away.

dif validate is the gate on the agent’s work

An agent will write a file that looks right. dif validate decides whether it is. Validation is collect-all rather than fail-fast, so one run reports every problem in the workspace:

{
  "ok": false,
  "errors": [
    {
      "code": "E005",
      "message": "variant weights sum to 90, expected 100",
      "file": "dif/experiments/active/pricing-annual-default.md",
      "line": 1,
      "column": 1,
      "help": "Distribute the variants so the weights total 100."
    }
  ],
  "warnings": []
}

Three codes catch most of what an agent gets wrong. E005 is arithmetic: weights must total exactly 100. E006 fires when an audience predicate uses an attribute that is not declared in dif/config.yaml, which is what happens when the agent invents country: US on a project that only declares locale and device_type. E007 fires when two active experiments share a surface without a shared exclusion_group and without provably disjoint audiences, which is the collision that would otherwise show some users a mashup of two tests and make the metric unattributable.

There is also W001, a warning that a scanned dif("id", ...) call site references no active experiment. That is the one to watch after a test ships, because the call site outlives the experiment file unless someone removes it.

Run dif validate --json in CI and the agent’s file fails the PR the way a type error does. That check is also the reason you can let an agent author experiments without reading every line of YAML yourself.

You read the result, and so does the next session

Concluding is a human verb. Say the test runs its 14 days and the annual default comes out ahead. You write the decision line:

dif conclude pricing-annual-default \
  --decision "Shipped variant_a. +2.4% paid_conversion over 14d, refund_rate flat."

Four things happen atomically. The file moves to dif/experiments/concluded/2026-09-pricing-annual-default.md. Its frontmatter gets status: concluded and today’s date. The ## Decision block fills with your text. And one dated line appends to dif/surfaces/pricing.md:

## Learnings

- 2026-09-28 pricing-annual-default: shipped variant_a. +2.4% paid_conversion over 14d, refund_rate flat.
- 2026-08-02 pricing-tier-order: killed. No effect on paid_conversion.

That log is the part that changes the next session. dif new embeds the surface’s last three learnings into every new draft as an HTML comment, and dif build regenerates dif/context.json with the current active set. Ask either tool six months later to test the plan toggle again and it reads that line first and tells you the question was answered. The mechanics are in what context.json is.

Write the decision like the next person reads it cold. “Inconclusive.” teaches nobody anything, and it is the line a future draft will quote back at you.

Where experimentation with Claude stops

The agent does not create traffic. A 50/50 test on 500 daily users needs the days a t-test needs, and no amount of tooling shortens that. What it compresses is the authoring and the readback.

It also should not conclude. Picking a winner changes what customers see, and an agent shown an early p-value will pattern-match on it. Day three at +2.4% and day fourteen at +0.3% are the same experiment.

Watch the hypothesis, too. Ask for an experiment without stating what you believe and the agent will write a plausible-sounding hypothesis to fill the field, which is worse than leaving it blank because it survives review. Write that line yourself and paste it into the prompt.

One build-level gotcha: dif/generated/ is gitignored, so CI and deploy builds have to run dif build before the app build. The usual fix is a "prebuild": "dif build" script in package.json. An agent that adds a call site without that in place ships a build error, not a broken experiment, which is the right direction to fail.

And the standing tradeoff: turning something off is editing a weight to 0 and merging. If your build takes ten minutes, your worst case is ten minutes. For a payment path that needs a sub-second kill, use something else for that flag.

FAQ

Can Claude Code run an A/B test end to end? It runs every part except traffic and the decision. It drafts the experiment file, writes both variants at the render site, runs dif validate until it passes, and opens the PR. dif buckets users and records exposures. You write the hypothesis up front and run dif conclude at the end.

Does this work in Cursor, or only in Claude Code? Both, with one difference. Claude Code gets three skills under .claude/skills/ that load on demand. Cursor gets .cursor/rules/dif.mdc, which attaches when the task touches dif/**. Anything else that reads AGENTS.md, including Copilot through .github/copilot-instructions.md, sees the same guidance.

What does dif init actually write into my repo? A dif/ directory (experiments, surfaces, audiences, config.yaml), plus the agent files: a managed block in CLAUDE.md, AGENTS.md, and .github/copilot-instructions.md, the three skill directories, and .cursor/rules/dif.mdc. Co-owned files are merged, never overwritten. Use --agents to pick a subset.

Do I need dif Cloud to run an experiment with Claude Code? No. Assignment is a local hash of the user ID, so flags and splits work with nothing configured. Run dif init --events custom and you get exposure.ts and track.ts handlers you own, pointed wherever your analytics already live. dif Cloud is an optional hosted analysis layer.

How do I stop the agent re-running a test we already concluded? Conclude with a real decision line. dif conclude appends it to the surface’s Learnings log, dif new embeds the last three learnings into the next draft, and dif build puts the active set in context.json. The agent reads all three before it drafts.

Can the agent conclude the experiment for me? It can draft the decision line and propose it. Running dif conclude is a call about the business, so it stays with a person. The same split applies to ramping a winner to 100%.

Getting started

Experimentation with Claude Code and Cursor is what you get when an experiment is a file instead of a dashboard row. The agent authors the Markdown and both variants because the schema is a shape it can fill in and the render site is code it can read. dif validate catches the weights, the undeclared attributes, and the surface collisions before review. You supply the hypothesis, read the result, and run dif conclude, which writes the learning that the next session starts from.

Install the CLI and scaffold the repo:

npm install -g @dif.sh/cli
dif init

That writes the dif/ workspace, the skills, the Cursor rule, and the managed blocks in CLAUDE.md and AGENTS.md. Then run dif build once so context.json exists, and ask your agent for the first experiment. The full loop from draft to decision is in agent-driven experiments.