Checking an experiment exclusion graph at build time, in Rust
dif is a CLI that reads one Markdown file per A/B test and compiles the active ones into a TypeScript module. The engine is a Rust crate, dif-core, about 5,000 lines including tests. This post walks through one part of it: how dif build decides whether two experiments could put the same user in both, and what it does when it cannot rule that out.
Two experiments on the same part of a page are a problem when one user is in both, because neither result can be attributed. Hosted experimentation tools such as Statsig and Eppo handle this with layers configured in their console. We wanted the check to fail in CI, before anything ships.
All code below is from cli/crates/dif-core/src/ in dif-sh/dif at v0.6.3, with some comments trimmed. All terminal output is from a release build of that tag.
Key Takeaways
- The graph is implicit. Nodes are experiments with
status: active, and an edge is a pair on the samesurfacethat might share a user.detect_conflictsinexclusion.rsenumerates the pairs per surface.- An edge is discharged two ways: both files declare the same
exclusion_group, oraudiences_disjointproves theincludesets cannot intersect. Any edge left over is errorE007anddif buildwrites nothing.- Output is deterministic by construction:
BTreeMapfor grouping, an explicit sort before every emit, a hand-writtenStringwriter, and no timestamp inclient.ts.- Bucketing is
SHA-256, four bytes, mod 10,000. One JSON fixture with 60 cases pins the Rust and TypeScript implementations to the same numbers.- The honest part: the
ExclusionGraphstruct is only called from tests, every non-YAML diagnostic points at line 1, and group order is re-sorted at runtime in the SDK.
The input: frontmatter into a serde struct
An experiment file is YAML frontmatter plus a Markdown body. Here is one of three files we will use, dif/experiments/active/hero-layout.md:
---
id: hero-layout
status: active
owner: dev@example.com
surface: home
hypothesis: >
A stacked hero lifts mobile signups.
audience:
include:
- device_type: mobile
variants:
- id: control
weight: 50
- id: variant_a
weight: 50
metrics:
primary: signup
created: 2026-09-15
---
The other two are hero-copy (created 2026-09-01, no audience) and nav-search (created 2026-09-20, include: device_type: [tablet, desktop]). All three target surface: home.
We did not pull in a frontmatter crate. split_frontmatter in parse.rs is about 40 lines that scan for the closing --- and return a &str slice of the YAML plus the byte offset where the body starts. The slice goes to serde_yaml::from_str and lands in this struct (spec.rs, doc comments removed):
#[derive(Debug, Clone, Serialize, Deserialize)]
pub struct Experiment {
pub id: String,
pub status: Status,
pub owner: String,
pub surface: String,
pub hypothesis: String,
#[serde(default)]
pub audience: Audience,
pub variants: Vec<Variant>,
pub metrics: Metrics,
#[serde(default)]
pub exclusion_group: Option<String>,
pub created: NaiveDate,
#[serde(default)]
pub concluded: Option<NaiveDate>,
}
Audience predicates are the one place we kept YAML untyped:
#[derive(Debug, Clone, Serialize, Deserialize)]
#[serde(transparent)]
pub struct AttrPredicate(pub serde_yaml::Mapping);
A predicate is a map from attribute name to a scalar (equality) or a sequence (membership). Keeping it as a Mapping means serde_yaml::Value does the comparison work later, since it implements Eq and Hash.
Loading is tolerant. Workspace::load in workspace.rs returns Err for two things: no dif/config.yaml in any ancestor directory, or a config that cannot be read or parsed. A broken experiment file becomes an E001 diagnostic in Workspace::parse_errors and loading continues, so one run reports every broken file.
The graph is a pairwise walk
There is no adjacency list. detect_conflicts groups active experiments by surface and walks every pair:
let mut by_surface: BTreeMap<&str, Vec<&ParsedExperiment>> = BTreeMap::new();
for p in &workspace.active {
if matches!(p.spec.status, Status::Active) {
by_surface.entry(p.spec.surface.as_str()).or_default().push(p);
}
}
for i in 0..exps.len() {
for j in (i + 1)..exps.len() {
let a = exps[i];
let b = exps[j];
// Shared exclusion_group → declared intent, runtime resolves.
let shared_group = matches!(
(&a.spec.exclusion_group, &b.spec.exclusion_group),
(Some(x), Some(y)) if x == y
);
if shared_group {
continue;
}
// Provably-disjoint audiences → can't collide on the same user.
if audience::audiences_disjoint(&a.spec.audience, &b.spec.audience) {
continue;
}
// ... push a Conflict with the two ids in lexical order
}
}
That is quadratic per surface. We are fine with it: a surface with 30 concurrent experiments is 435 pairs. On the three-file workspace here, dif build in release mode finishes in under 10 ms on an Apple M4, and that includes walking the repo for call sites.
The BTreeMap matters more than the complexity. Iteration order decides the order of diagnostics, and a HashMap would shuffle them between runs.
Proving two audiences disjoint
audiences_disjoint in audience.rs is a deliberately small prover:
pub fn audiences_disjoint(a: &Audience, b: &Audience) -> bool {
let a_attrs = include_constraints(a);
let b_attrs = include_constraints(b);
for (attr, a_values) in &a_attrs {
let Some(b_values) = b_attrs.get(attr) else {
continue;
};
if sets_disjoint(a_values, b_values) {
return true;
}
}
false
}
include_constraints collects, per attribute, the union of every value the audience’s include list names. If both audiences constrain the same attribute and the two value sets share nothing, a user holds one value for that attribute and cannot match both.
Returning false means “could not prove it”, not “they overlap”. The function only errs in that direction. An audience that lists the same attribute in two include entries is really an AND, but we take the union, which is a superset and so can only make the proof harder to reach.
What it prints
With no exclusion_group anywhere:
$ dif build
error E007 dif/experiments/active/hero-copy.md:1:1: experiments `hero-copy` and `hero-layout` both target surface `home` without a shared exclusion_group, and their audiences are not provably disjoint
help: Add the same `exclusion_group:` value to both, narrow one of the audiences so they're provably disjoint, or change one of their `surface:` fields.
error E007 dif/experiments/active/hero-copy.md:1:1: experiments `hero-copy` and `nav-search` both target surface `home` without a shared exclusion_group, and their audiences are not provably disjoint
help: Add the same `exclusion_group:` value to both, narrow one of the audiences so they're provably disjoint, or change one of their `surface:` fields.
✗ build refused: 2 error(s)
fix the above and re-run, or run `dif validate` to recheck.
$ echo $?
1
Three experiments, two errors. The hero-layout and nav-search pair is missing because mobile and [tablet, desktop] do not intersect.
Adding exclusion_group: home to hero-copy and hero-layout removes one error and leaves the hero-copy and nav-search pair. Edges are pairwise, so a group has to be declared on every member. With all three in the group, validation passes.
build.rs in dif-cli runs validate::run first and returns ExitCode::from(1) before any codegen function is called. Validation is collect-all: twelve passes push into one Report with errors and warnings vectors, and sort_report orders both by file, line, column, code and message. Passing --json serializes the same Report through serde.
What the build emits
$ dif build
✓ validated 3 active experiment(s)
✓ typed client → dif/generated/client.ts
✓ audiences → dif/generated/audiences.ts
✓ events → dif/generated/events.ts
✓ context → dif/context.json
One entry from client.ts:
__register({
id: "hero-layout",
surface: "home",
variants: ["control", "variant_a"] as const,
salt: "c61c93451586360d4e1ab78144f42024",
weights: { "control": 50, "variant_a": 50 },
exclusionGroup: "home",
created: "2026-09-15",
audience: (attrs) => {
if (attrs["device_type"] !== "mobile") return false;
return true;
},
});
The YAML predicate is compiled to an arrow function with one early return per constraint. The salt is precomputed, so the SDK never hashes the experiment id. There is no JSON blob to parse at startup and no FFI, NAPI or WASM between the Rust side and the JavaScript side. The contract is this file.
codegen.rs builds the file with String::push_str and format!. We did not use a template engine or run a formatter over the result. Experiments are sorted by id before rendering, and the banner carries the crate version but no timestamp. We ran dif build twice a second apart and shasum -a 256 printed the same digest for client.ts both times. dif/context.json is different on purpose: it carries generated_at.
Resolution at runtime, and replaying it in Rust
A shared group tells the validator the pair is handled. The handling is a priority order: sort members by (created, id) and the first one whose audience matches the user wins. exclusion.rs models the outcome as an enum:
#[derive(Debug, Clone, PartialEq, Eq)]
pub enum Outcome {
Assigned { variant: String, bucket: u16 },
Forced { variant: String },
AudienceMiss,
ExclusionLoser { winner: String },
}
resolve uses one BTreeMap for grouped and ungrouped experiments alike. An ungrouped experiment gets a private key that no YAML string can equal:
let key = match &p.spec.exclusion_group {
Some(g) => g.clone(),
None => format!("\0solo::{}", p.spec.id),
};
dif qa calls resolve and prints the rows:
$ dif qa --user u_15 --attr device_type=mobile
trace u_15:
• hero-copy → control (bucket 4886)
• hero-layout ↛ exclusion loser (winner: hero-copy)
• nav-search ↛ audience miss
The TypeScript SDK implements the same rule in exclusionGroupWinner in cli/packages/sdk/src/core.ts, using the created and exclusionGroup fields from the generated file.
Bucketing, briefly
bucket.rs is 98 lines. The salt is the first 16 bytes of SHA-256("dif.sh/v1" || experiment_id), and the bucket is:
let quad = u32::from_be_bytes([digest[0], digest[1], digest[2], digest[3]]);
(quad % 10_000) as u16
An earlier version read two bytes. 2^16 % 10_000 is 5,536, so buckets 0 to 5,535 had seven preimages and the rest had six, which made a nominal 50/50 split come out near 53.4/46.6. A test now hashes 100,000 synthetic ids and asserts the split lands within one point of even. The bucketing post covers the math in full.
tests/bucket_fixture.rs writes bucket_tests.json from the Rust implementation: 6 experiment ids by 10 user ids, including an empty string and multi-byte UTF-8. The TypeScript test suite loads the same file and must reproduce all 60 buckets. Setting DIF_REGEN_FIXTURE=1 regenerates it, and the diff shows up in review.
Testing
cargo test at v0.6.3 runs 185 tests: 102 unit tests in dif-core, 82 in dif-cli, and the fixture test. Nearly all of the core tests go through parse_experiment_str, the IO-free entry point, with frontmatter built by string replacement on a constant. A Workspace is a plain struct with public fields, so the validator and resolver tests construct one directly without touching a disk.
Dependencies in dif-core are serde, serde_json, serde_yaml, thiserror, miette, sha2, chrono, walkdir and regex. The CLI adds clap, console, indicatif and dialoguer.
What we would do differently, and open questions
The graph struct is not on the build path. ExclusionGraph::build exists, sorts each group by priority, and has two tests. Nothing in dif build calls it. The SDK filters and sorts the registry on each assignment instead. Emitting the sorted member list per group from Rust would make the name of this module true and remove a sort from the hot path.
Two resolvers, one fixture. The bucket math is pinned across languages. The group arbitration is not. pick_winner in Rust and exclusionGroupWinner in TypeScript are kept in sync by comments that reference each other. A shared resolution fixture is the obvious next test.
Diagnostics have no spans. Diagnostic derives miette::Diagnostic, but the CLI formats lines by hand and simple_error hard-codes line 1, column 1. E007 is anchored to the lexically first file of the pair only. YAML errors do carry a location, but it is relative to the frontmatter slice, so a bad weight on file line 14 is reported as 13:13. We need key positions threaded through the parser.
The prover ignores exclude. include: device_type: mobile next to exclude: device_type: mobile is disjoint, and we report E007 anyway.
Priority can starve. In the trace above hero-copy has no audience and the earliest date, so it wins the group for every user and the other two get no traffic. That is correct per the rule and surprising in practice. Should the validator warn when an unrestricted member shadows the rest, or should a group split traffic by hash? We have not decided.
serde_yaml 0.9 is archived upstream. We need to move, and audience values typed as serde_yaml::Value make that a wider change than it should be.
The call-site scan is a regex. scan_call_sites matches dif("id" in .ts, .tsx, .js and .jsx files. It also matches the same text inside a comment.
Writes are not atomic. The four output files are written one after another with std::fs::write.
The source is MIT licensed at github.com/dif-sh/dif under cli/crates/. The crates are not on crates.io, so building means cd cli && cargo build --release. If you have built a disjointness check like this over a richer predicate language, we would like to hear where it stopped being simple.