Using dif skills for agent-driven experimentation

dif init installs more than a folder. It drops three Claude Code skills into .claude/skills/ and writes a guidance block into your agent config files, so a coding agent can draft, edit, and conclude experiments next to you. The skills read dif/context.json and the surface logs on the way in, so prior learning travels with the work.

What dif init put in your repo

Alongside the dif/ workspace, dif init writes:

.claude/skills/
  dif-author-experiment/     # draft and iterate on an experiment
  dif-conclude-experiment/   # write the decision, archive, learn
  dif-generate-surfaces/     # propose the surface set from the codebase
CLAUDE.md                    # a managed dif block
AGENTS.md                    # the same block, for other agents
.cursorrules                 # the same block, for Cursor

dif owns only its marked block in those three files, so anything else you keep there survives a re-run. To skip all of it, pass dif init --no-agent-files. The CLI, the SDK, and the build work the same without it.

context.json: the session-start read

dif build regenerates dif/context.json. It lives outside the gitignored generated/ folder, so it is committed, and any agent that opens the repo sees the current state. Think of it as a CLAUDE.md for your experiments:

{
  "active": [
    { "id": "checkout-cta-v2", "surface": "checkout",
      "variants": ["control", "variant_a"], "running_for_days": 7 }
  ],
  "surfaces": [
    { "name": "checkout",
      "recent_learning": "Get it today lifted conversion 2.1% on returning" }
  ],
  "conventions": ["Fire exposure at render, never at assignment."]
}

The file carries what is active, on which surface, for how long, each surface's most recent learning, and the project conventions. Because the skills read it first, a new draft starts from what the last test found instead of from a blank page.

The three skills

Each skill is a plain SKILL.md with a trigger description. Your agent loads one when your request matches: say "draft an experiment" and dif-author-experiment answers. They follow the daily loop, and the skills reference covers each one in full.

dif-author-experiment

For drafting, editing, or fixing an experiment. It reads three files before writing anything: dif/context.json for what is already running, the surface's ## Learnings log so it does not re-test a settled question, and dif/config.yaml for the audience attributes it may use. Then it runs the loop:

$ dif new <id> --surface <surface>
# fill in the hypothesis, brief, rationale, audience, metrics
$ dif validate
$ dif build

It fills the stubs with opinions: one falsifiable hypothesis, variants that total 100 with control first, a single primary metric, and predicates only over declared attributes.

dif-conclude-experiment

For closing a test that has run. The decision you write becomes the next line in the surface log, and the next draft on that surface reads it. So the skill will not conclude until it can answer four questions: is there a real analysis of the primary metric, did the test run long enough on the right traffic, does the outcome fit one sentence with a direction and a number, and did the guardrails hold.

$ dif conclude checkout-cta-v2 \
    --decision "Shipped variant_a. +2.1% completed-checkout on returning, p<0.01 over 14d. Guardrails clean."

That command moves the file to concluded/, fills the ## Decision block, and appends a dated line to the surface log. A useful decision reads like the one above. "Inconclusive." tells the next draft nothing.

dif-generate-surfaces

For a fresh repo, where dif init left a single surface. A surface is a place you would run a test, not a route or a component, so most apps have three to eight. The skill reads your README, your routes, and your config, proposes a list with a one-line reason for each, and waits for your yes before it writes the files.

A session looks like this

You talk to the agent the way you would a teammate. To start a test:

Draft an experiment on checkout testing a more urgent CTA, for returning visitors.

It reads the checkout log, runs dif new, fills the frontmatter, and validates. When the numbers are in:

Conclude checkout-cta-v2. Variant A won, +2.1% over two weeks.

It writes the decision, archives the file, and appends the learning. You review the diff and commit, the same as any other change.

The skills are an accelerant, not a requirement. The CLI and SDK behave the same without them, and the AGENTS.md and .cursorrules blocks point non-Claude agents at the same workflow. Once a test ships, the next step is a team-wide live read, which is what Dif Cloud adds.