Validate feature flags in CI with dif validate

Validate feature flags in CI by running dif validate on every pull request. The command is a type checker for your flag files: it fails the PR before a broken flag lands on main, the way a failed test fails a broken code path.

One pass checks that variant weights total 100, the owner field is a real email, and every referenced surface and attribute exists in the repo. It scans your app source for dif("...") call sites and warns on orphans. It catches two active experiments colliding on one surface with diagnostic E007. Any failure exits non-zero, and CI fails the PR.

This post covers what each check does, what breaks when you skip it, a paste-able GitHub Action, and how a coding agent runs the same command on its own work before opening a PR. It is the CI half of the git-native flag workflow.

Key Takeaways

dif validate is a single-pass check on your flag files: variant weights sum to 100, owner is a valid email, referenced surfaces and attributes exist, and no two active experiments collide on one surface (E007).

It scans app source for dif("...") call sites and warns on orphans, so a flag nobody deletes stays visible in review instead of accumulating.

Run it in CI on every pull request. A broken flag file fails the PR like a broken build.

Failure modes in dif always resolve to the first variant and record nothing, but that only saves you at runtime. CI catches the broken file before it ships.

The tradeoff: CI validation catches config errors, not intent errors. A valid flag can still gate the wrong code path.

What `dif validate` checks

dif validate runs a few checks in one pass. Each one maps to a class of bug that used to be caught by an incident, or a stale PR comment, or nothing at all.

Weights sum to 100. Every variant in a flag has a weight. Two variants at 50/50 total 100. A rollout at 10/90 totals 100. A typo like weight: 5 and weight: 90 totals 95, so five percent of the bucket space maps to no variant. The validator catches that in the file before the assignment math ever runs.
Owner is a valid email. Every flag file has an owner. A missing owner, or sam with no domain, fails the check. An owner is the answer to “who do I ask about this flag” three months from now.
Referenced surfaces and attributes exist. A flag with surface: chekout (typo) fails because dif/surfaces/chekout.md does not exist. An audience filtering on device_typ fails because your audience schema does not have that attribute. Both would silently degrade at runtime.
Orphan dif("...") call sites. dif scans your app source for calls to dif("some-flag") and warns if the flag file is missing. It also warns on the reverse: a live flag file with no call site anywhere in the code. That is the flag graveyard problem, made visible in the diff.
Collision detection (E007). Two active experiments on the same surface must share an exclusion_group or have provably disjoint audiences. If neither is true, a user could land in both experiments at once, and there is no clean way to attribute the metric movement. The CLI reference documents the diagnostic codes.

On a clean repo the output is short:

$ dif validate
Checking dif/experiments/active/checkout-cta-copy.md ... ok
Checking dif/experiments/active/new-checkout.md ... ok
Checking dif/surfaces/checkout.md ... ok
Checking dif/surfaces/home.md ... ok
Checking dif/audiences/mobile.md ... ok

Scanning src/ for dif() call sites ... 8 calls, 0 orphans
Checking active experiment collisions ... 0 collisions

All checks passed.

On a broken repo it tells you what to fix:

$ dif validate
Checking dif/experiments/active/checkout-cta-copy.md ... FAIL
  variant weights total 95, expected 100
    control:     weight 5   (was 50?)
    benefit_led: weight 90

Checking dif/experiments/active/new-checkout-copy.md ... FAIL
  E007: collision on surface `checkout`
    active experiments on this surface:
      - checkout-cta-copy   (audience: device_type in [mobile, tablet])
      - new-checkout-copy   (audience: device_type in [mobile])
    audiences overlap and neither declares exclusion_group

Scanning src/ for dif() call sites ... 8 calls, 1 orphan
  orphan: dif/experiments/active/old-hero-test.md
    referenced by: none

2 flags failed. See https://dif.sh/docs/troubleshooting/ for diagnostic codes.

Why validate feature flags in CI when the SDK will surface errors at runtime?

dif’s runtime is fail-safe by default. Every failure mode (broken audience, missing variant, dangling reference) resolves to the first variant and records nothing. That protects your users. It does not protect your data.

Ask what happens when a bad flag file merges to main. The build succeeds. The site deploys. Users hit the render site, get the first variant, and never see the broken one. Nothing looks wrong. But the experiment is not running. Every user is being force-assigned to control. You will not notice until the results panel shows zero exposures for variant_a, or worse, shows a split that does not match the file.

Validating in CI catches this before the branch ever merges. It is the same argument as running a type checker in CI: the language would still run the code at runtime, and it would still crash, but a type error caught in CI is cheaper than a stack trace in production.

What breaks when an invalid flag file lands on main

Three failure modes are common, and each one is quiet in production.

Weights that do not sum to 100. A typo turns 50/50 into 5/90, so five percent of the hash space maps to no variant. dif’s runtime resolves the unmapped bucket to the first variant, so nobody sees a broken page, but the split is now 55/45 and not what the file says. Any conclusion you draw from that experiment is against a split you did not actually run.

A reference that does not exist. A flag with surface: chekout silently degrades: the surface log never updates, because there is no chekout.md to write to. The next dif conclude writes to a file nobody will look for. The learning is real, the record is lost.

A collision on one surface. Two active experiments target device_type: mobile on checkout, and neither declares an exclusion_group. A mobile user is in both. The completed_checkout metric moves. You cannot say which experiment moved it. dif validate catches this with E007 in review, so it never gets to production.

The point is not that dif crashes without the check. It does not: fail-safe by default means the worst runtime outcome is control. The point is that the data you thought you were collecting is not the data you actually collected.

A GitHub Action for validating feature flags in CI

Here is a complete .github/workflows/dif-validate.yml that runs dif validate on every pull request. Paste it into any repo that has dif init run in it.

name: dif validate

on:
  pull_request:
    paths:
      - 'dif/**'
      - 'src/**'
  push:
    branches:
      - main
    paths:
      - 'dif/**'
      - 'src/**'

jobs:
  validate:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4

      - name: Set up Node
        uses: actions/setup-node@v4
        with:
          node-version: '20'

      - name: Install dif CLI
        run: npm install -g @dif.sh/cli

      - name: Validate flags
        run: dif validate

Three things worth noting. The paths filter runs the job only when a flag file or app source changes, so unrelated PRs stay fast. The CLI installs globally with the same command a developer runs locally, so a green CI check means the same code passed on both sides. dif validate exits non-zero on any failure, which GitHub Actions treats as a failed step, which fails the required check on the PR.

Make the job required in branch protection, and a broken flag file cannot merge.

How a coding agent uses `dif validate` locally

When a coding agent generates a flag, it should check its own work before opening the PR. dif init writes managed blocks into AGENTS.md, CLAUDE.md, and .cursorrules telling the agent that dif ships a validator and to run it after authoring or editing a flag. A short session looks like this:

User: Gate the new checkout button behind a 10% mobile rollout.

Agent: I'll write the flag file and run dif validate.

$ cat > dif/experiments/active/new-checkout-button.md <<EOF
---
id: new-checkout-button
status: active
owner: agent
surface: checkout
audience:
  include:
    - device_type: [mobile]
variants:
  - id: "off"
    weight: 90
  - id: "on"
    weight: 15
---
EOF

$ dif validate
Checking dif/experiments/active/new-checkout-button.md ... FAIL
  variant weights total 105, expected 100 (off=90, on=15)
  owner "agent" is not a valid email

Agent: Weights add to 105, not 100, and the owner needs a real email.
Fixing both.

$ dif validate
Checking dif/experiments/active/new-checkout-button.md ... ok

Scanning src/ for dif() call sites ... 9 calls, 0 orphans
All checks passed.

Agent: Green. Opening the PR.

The agent reads the error, edits the file, and reruns. Nothing about that requires a human at the terminal. The same command that runs in CI is the one the agent runs locally, so the PR that arrives is one an AI agent already validated.

dif validate catches config errors, not intent errors. It knows the weights sum to 100. It does not know the agent gated the wrong button. Code review still owns that call.

FAQ

What does dif validate check? A single pass over the flag files in your repo. It confirms variant weights total 100, the owner field is a valid email, referenced surfaces and attributes exist, no orphan dif("...") calls in the app source, and no two active experiments colliding on one surface (E007).

Why run dif validate in CI instead of trusting the runtime? The runtime is fail-safe: a broken flag resolves to the first variant and records nothing, so nobody sees a broken page. That protects your users but not your data. An experiment silently force-assigning every user to control looks fine in production and useless in the results.

Does dif validate need dif Cloud? No. The CLI is free and open source. dif validate runs against the files in your repo with no account and no network call.

What happens when the validator fails a PR? dif validate exits with a non-zero code. The GitHub Actions step fails, the check on the PR turns red, and branch protection blocks the merge until the flag file is fixed and the check passes.

Can I validate a single flag instead of the whole repo? dif validate checks every active flag by default, because collisions and orphans need the full picture. Full sweeps still belong in CI.

What about intent errors, like the flag gating the wrong code path? The validator does not catch those. A valid flag can still gate the wrong branch. Code review and dif qa (trace a user’s assignment locally, without a deploy) are the tools for that.

Getting started

Validating feature flags in CI moves a class of error from production to the pull request, where it is cheap. dif validate runs on the files in your repo, checks the five things that cause silent data drift, and exits non-zero when it finds one. Wire it into a GitHub Action and a broken flag cannot merge.

For teams working with coding agents, the same command is what the agent runs locally on its own output. The PR that arrives has already been checked against the schema. Reviewers spend their time on the code path, not the frontmatter.

Install the CLI and scaffold a project:

npm install -g @dif.sh/cli
dif init

Then add the workflow above and make it a required check. See the dif docs for the rest of the workflow.