What an experimentation platform actually requires
An experimentation platform needs five things to run tests a team can trust: deterministic assignment, protection against two tests colliding on the same users, validation before a change ships, a record of what was decided and why, and a failure mode that never makes things worse than not shipping. Most tools give you a dashboard and a p-value and call that a platform.
A dashboard shows a list of running tests. It does not stop two engineers from targeting the same checkout page with conflicting experiments, and it does not remember why a test got killed after whoever ran it leaves the company. The gap between “has an A/B testing UI” and “is a platform a team can trust” is where most vendor pitches go vague, because the hard parts are not in the chart.
This post names the five requirements and the mechanism each one needs, using dif’s git-native model as the worked example, its own tradeoff included.
Key Takeaways
- An experimentation platform needs deterministic assignment (no stored lookup table), so a user never flickers between variants across page loads or devices.
- Collision protection matters as much as targeting: two active tests on one surface must share an exclusion group or have provably disjoint audiences, or the metric movement is unattributable.
- Validation belongs in CI, not a support ticket after launch. A weight that does not sum to 100, or a reference to a surface that does not exist, should fail the build.
- A decision log beats a dashboard. Every concluded test needs a dated decision and a one-line learning the next test on that surface can read, or the same idea gets re-tested two years later.
- A platform’s failure mode is a requirement, not an edge case. Every miss (no user id, unknown flag, audience miss) should resolve to the default and record nothing.
Assignment an experimentation platform can trust
The first requirement is boring and non-negotiable: the same user has to land in the same variant, every time, on every device, without a database lookup slowing down the request.
The fix is deterministic, hash-based assignment. Take the user id, hash it with a per-experiment salt, and reduce the result into a fixed number of buckets. dif takes the first four bytes of SHA-256(salt || user_id), mod 10,000, and reads the variant off the cumulative weights. Nothing is stored. The same input produces the same bucket every time, on the server or in the browser, with no network call to decide.
That buys three properties an experimentation platform needs and a lookup table does not: no flicker across page loads, additive ramps (widening a rollout only repaints more buckets, so nobody has the new thing taken away mid-flight), and independence (a separate salt per test means being in the 10% for one does not correlate with your bucket in another).
Why two tests collide on the same surface
The second requirement shows up the first time two teams ship experiments on the same page in the same sprint. If a checkout-copy test and a checkout-layout test both target 50% of users with no coordination, some users land in both, and nobody can attribute the metric movement to either one.
An experimentation platform needs a way to guarantee mutual exclusion on a shared surface. dif calls this an exclusion group: tag two files with the same exclusion_group key, and each user participates in at most one. Members are ordered by their created date, the first matching audience wins, and the rest show their default and record nothing.
---
id: checkout-copy-test
status: active
owner: sam@acme.com
surface: checkout
exclusion_group: checkout-copy
created: 2026-09-02
variants:
- id: control
weight: 50
- id: benefit_led
weight: 50
metrics:
primary: completed_checkout
---
Without this, a platform’s own validator should catch the collision before it ships, not after an incident review tries to untangle which test moved the number.
Validation that fails the build, not the user
Most tools let a broken configuration reach production: a variant weight that does not sum to 100, an owner field that is not a real email, a reference to a surface that was renamed last quarter. An experimentation platform needs to catch these the same way a type checker catches a bad function signature, before a reviewer approves anything.
dif validate runs one pass over every active file: weights must total 100, owners must be valid emails, referenced surfaces and audience attributes must exist. It scans the application source for dif("...") call sites and flags orphaned references, and it runs the collision check from the section above as diagnostic E007.
$ dif validate
✗ checkout-copy-test.md: E007 collides with checkout-layout-test.md
on surface "checkout" with no shared exclusion_group
✗ pricing-cta.md: weights sum to 90, expected 100
2 errors, 0 warnings
Run it through the CLI reference in CI, and a broken experiment fails the pull request the same way a broken test suite does. That is cheaper than finding the collision in a dashboard after the metric already moved.
Validation catches a malformed test. It does not catch a true one read wrong, which is a discipline problem more than a tooling one. Evan Miller’s write-up on how not to run an A/B test explains why checking the result early and stopping the moment it looks significant inflates false positives. Microsoft’s research on trustworthy experimentation is the standard reference for picking a primary metric and a guardrail metric before a test starts, not after a win already looks good. A platform can enforce the file format. It cannot enforce patience.
A decision log, not a dashboard that forgets
A platform that only shows active tests has already lost the thing that matters most: what got decided, and why. Dashboards list what is running. They do not carry forward what the team already learned, so the same disproved idea gets proposed again by someone new in two years.
An experimentation platform needs a decision log attached to the part of the product being tested, not to the test itself. In dif, a surface (checkout, pricing, home) is a file that accumulates a dated, one-line record of every concluded experiment on it. dif conclude writes the decision and the learning back to the surface and archives the test file. The next dif new on that surface pre-loads the last three learnings into the draft, so “didn’t we already try this” is answered by a file instead of someone’s memory.
That log is also the artifact a coding agent can read. dif build emits context.json, a summary of every active flag and each surface’s latest learning, and dif init writes managed blocks into CLAUDE.md and AGENTS.md so an agent authoring a new test starts from what already happened instead of guessing.
Fail-safe defaults for every miss
The fifth requirement is the one marketing pages skip: what happens when the platform itself fails. No user id on the request. An unknown flag id. A variant renamed in the file but not yet at the call site. An audience miss.
Every one of those has to resolve to the default variant and record nothing. Not “best guess,” not an error page, nothing. The worst case of a misconfigured experiment should be that nobody sees the new thing, not an incident. This also keeps the data clean: an unassigned user is never counted as if they saw a variant, so a failure mode cannot quietly bias a result.
A platform that cannot state its failure mode in one sentence has not defined one.
What a real experimentation platform gives up
LaunchDarkly and Optimizely win two rows outright against a git-native model: an instant, sub-second kill switch, and broader SDK coverage across backend languages. dif does not close that gap and does not pretend to.
The cost is real. Rolling a test back in dif means editing a number and merging, so the kill time is however long the build takes. If your build takes ten minutes, the worst case is ten minutes. For a payment path that needs a kill in under a second, that is the wrong tool for that one flag, and a hosted platform with a push-based kill switch is the right one. For most experiments, where the failure mode is “some users see the new copy for a few more minutes,” a merge and a deploy is fine, and every miss still resolves to the default.
The trade is a repo instead of a dashboard: the test is reviewed in a pull request, its history is the git history, and the decision lives in the same file as the hypothesis, permanently.
Running an experimentation platform without the dashboard
You do not need a hosted dashboard to meet these five requirements. You need deterministic assignment, collision protection, validation in CI, a decision log, and a defined failure mode, and those can all live as files next to the code they test.
npm install -g @dif.sh/cli
dif init
That scaffolds a dif/ directory and a generated client. There is no account to create and no API key to copy. dif new drafts an experiment pre-loaded with that surface’s recent learnings, dif validate checks it, and dif build compiles it into a typed client. If you want the analysis layer too, dif Cloud is optional and separate from the CLI and SDK.
None of the five things an experimentation platform needs require a dashboard: assignment that does not flicker, a check that catches a collision before launch, and a file that still says what happened after the test is gone. For most teams, that beats a dashboard with no memory. Read the .md format or run your first experiment to see the whole lifecycle end to end.
FAQ
What is an experimentation platform? A system that assigns users to variants, tracks a metric, and reaches a decision about which variant wins. The minimum requirements are deterministic assignment, protection against two tests colliding on the same users, validation before launch, a record of the decision, and a defined behavior when something is misconfigured.
Do I need a database to run an A/B test? No. Assignment can be a pure function of the user id and a salt, recomputed on every request instead of looked up. dif hashes the id with SHA-256 and reduces it mod 10,000, so there is no assignment table to keep in sync and no network call to decide.
What’s the difference between a feature-flag platform and an experimentation platform? Mostly intent. A feature flag decides which code path runs; an experiment holds that decision at a fixed split with a hypothesis and a primary metric until the result is significant. In dif they are the same file format, so a flag can become an experiment and back without touching application code.
Can an open source experimentation platform replace LaunchDarkly or Optimizely? For most experiments, yes: deterministic assignment, CI validation, and git-based review cover the same ground, and the CLI and SDK are MIT licensed and free. The one place a hosted platform still wins is a sub-second kill switch on a critical path, where a merge-and-deploy rollback is too slow.
What happens when an experimentation platform’s assignment fails? It should resolve to the default variant and record nothing, every time, whether the cause is a missing user id, an unknown flag, or an audience miss. If a vendor cannot tell you that in one sentence, ask what actually happens, because it matters more than the uptime number on their status page.
Does an experimentation platform need its own dashboard? No. A dashboard is one way to see what is running, but the requirements (assignment, collision protection, validation, a decision log, fail-safe defaults) can live in files reviewed through the same pull request workflow a team already uses.