Assignment and Experiment Registry

Two campaigns launch in the same week. Both randomize properly. The lifecycle test reports a lift, the pricing test reports nothing, and both readouts go into the same deck.

Three weeks later someone notices that a large share of the lifecycle treatment group was also in the pricing treatment. Now neither number means what it says, and there is no way to repair either one, because assignment was done with a random draw at send time and nobody wrote down who got what.

That is the failure this component prevents. Not by running fewer tests, but by making assignment a property of the person and the experiment rather than an event that happened once inside a job that has since been rerun.

The decision it encodes

Which arm is this customer in, and does that conflict with another live test?

The first half is mechanical and almost everyone gets it roughly right. The second half is an organizational decision that has to be made before the collision, and it is the one that decides whether a year of tests adds up to anything.

Assignment is a hash, not a stored coin flip

Assign by hashing the unit together with a per-experiment salt, then bucketing the result. The arm becomes something anyone can recompute at any time from two strings, rather than a row in a table that has to be kept alive.

That property does more work than it appears to. A backfill can be rerun without reshuffling anyone. A late-arriving customer lands in the arm they would always have landed in. A downstream job that lost its copy of the assignment table can reconstruct the whole thing. And an analyst asking “which arm was this person in during the September test” gets an answer six months later without an archive.

The unit is a modeling choice, not an implementation detail. It has to sit at the level where interference happens. Randomize by person when the treatment reaches a person, by account when several people share a subscription and talk to each other, by household when the outcome is a shared purchase, and by geo when the channel cannot be targeted more finely than that. Choosing a unit smaller than the interference is how a test comes back with a clean interval around a contaminated number. The registry records four units per experiment, because they differ: the contact unit the message is delivered to, the assignment unit the hash runs on, the outcome unit the result is counted at, and the interference unit the assignment has to contain.

The registry is where the organizational decision lives

Every live test claims its space in one place, and the claim records the unit, the salt, the audience version, the arms and their shares, the start and end dates, and its exclusivity class.

The exclusivity class is the part that matters. Two tests can relate in three ways.

Orthogonal. Independent salts, so assignment in one carries no information about the other. Interactions average out across both readouts. This should be the default, because it is the only relationship that scales past a handful of concurrent tests.

Mutually exclusive. The treatments interfere badly enough that no one may be in both, so they draw from disjoint slices of the same hash space. Correct, and expensive: two exclusive tests at 50% each consume the entire population.

Nested. One test runs inside an arm of another, when the question is explicitly conditional. Cleanest of the three to analyze and the rarest to see in practice.

Defaulting to orthogonal only works if you know how much overlap you are accepting. With independent assignment, that number is arithmetic rather than opinion.

Orthogonality does not make interference disappear. It makes each test’s estimate an average over whatever else was running, which is usually the quantity you want, because whatever else was running is also what will be running when you act on the result. The failure is not overlap. The failure is overlap you did not know about, on treatments that push the same behavior in opposite directions.

The outcome contract, frozen with the assignment

The registry records more than who is in which arm. Every experiment carries its outcome contract, written before launch and read by the readout rather than chosen at it: the primary outcome and its metric version; the assignment and outcome units; the horizon and the rule for when an outcome counts as mature; how an outcome becomes value, with the treatment and offer cost netted; the smallest effect worth acting on; the guardrails and their bounds; the analysis population; the treatment of missing and censored outcomes; the repeated-look procedure; and the planned contrasts. A subscription program’s contract might name incremental twelve-month contribution per eligible customer as the value, with ninety-day conversion as the early outcome beneath it and unsubscribes and complaints as guardrails. Percentage points on a conversion do not become a decision without that conversion to value, and the policy downstream cannot supply it for itself.

Priority, decided in advance

Sooner or later two campaigns want the same person and the treatments cannot coexist. Someone has to lose.

That decision gets made either by a rule written down beforehand or by whichever job happened to run first. Only one of those is auditable, and the second one also silently correlates arm assignment with job scheduling, which is a confounder nobody goes looking for.

A workable rule ranks claims by expected incremental value per contact, falls back to a declared campaign priority when that is not estimable, and records the loser rather than dropping it. The record is what lets you answer “why did this person not get the winback offer” without reconstructing a week of logs.

The program holdout

A per-campaign control measures a campaign. A program holdout measures the program. They are different randomizations and the registry keeps them apart: the campaign control is drawn from the customers eligible for that action at that decision, outside the program holdout, and the program holdout is never used as the comparison for an individual campaign, because its members have a year of not being contacted behind them and this week’s recipients do not.

A prespecified share of eligible customer units receives the baseline lifecycle policy instead of any discretionary action, for a fixed and long tenure. It is the only way to answer whether the whole apparatus of lifecycle marketing is worth what it costs, which is a question that gets asked roughly once a year, usually by someone who can cancel it.

The cost is real and should be stated plainly rather than buried: you are forgoing the effect on those people for as long as they stay in. Tenure is a genuine trade. Too short and you never observe long-run effects, because nobody stays untreated long enough for one to appear. Too long without new enrollment and the holdout stops resembling the population it stands for, since everyone in it is having a different experience of the brand. Rolling enrollment cohorts, assigned on entry and retained for a fixed tenure, keep it representative without the churn of rotating treated customers in and out.

The size question comes up immediately and has a cleaner answer than it seems to. Precision on a two-arm comparison depends on 1/n₀ + 1/n₁, so the smaller arm dominates: at ten times the holdout, the treated arm still widens the interval by about 5%, and past that it contributes almost nothing. The holdout alone sets what the program-level readout can detect, and enrolling more treated customers does not improve it.

That is why a persistent holdout can be small in percentage terms and still work. On a base of five million, a 1% holdout is fifty thousand people, which against a 20% annual outcome rate detects a 2.5% relative change at 80% power and a 5% two-sided test. Doubling the holdout to 2% takes that to 1.8%. The forgone revenue doubles and the detectable effect improves by under a third, which is the trade worth putting in front of whoever has to approve the holdout, in those terms, before they are asked to approve it.

Maturity, honestly staged

Stage 1: deterministic assignment with per-experiment salts. Arms become reproducible and reruns stop reshuffling people. No new capability, and it removes an entire class of result nobody can defend.

Stage 2: a registry every test must claim from. Unit, salt, audience version, arms, dates. Overlap becomes visible before the readout rather than after it.

Stage 3: declared exclusivity classes and a written priority rule. The collision decision moves from scheduling accident to policy.

Stage 4: a program holdout with its scope contract, and value-based priority. The second requires per-state incremental estimates to rank claims, so it depends on measurement and decisioning being in place. Last for good reasons rather than through neglect.

bucket = hash(salt + ”:” + unit_id) mod 10000
arm    = “treatment” if bucket < 10000 * share else “control”