When the Experiment Leaks

Experiments leak. A third of the treated group never received the message, a handful of control accounts were contacted by a rep who did not know, and the treated users told their colleagues.

These get discussed as one problem, usually under the word contamination, and treated with the same remedy, usually dropping somebody from the analysis. They are three different failures, they bias in different directions, and only one of them is beyond the reach of the standard defense.

Three leaks, and what each one does

Noncompliance. Assigned to treatment, did not receive it. An email that bounced, an ad that never served, a feature flag that failed. The treated arm contains untreated people, so the measured difference is diluted toward zero.

Contamination. Assigned to control, received it anyway. A rep contacts a holdout account, an operational message goes to everybody, a person in a dark market sees the ad while traveling. The control arm contains treated people, so the difference is diluted toward zero again, from the other side.

Spillover. Treating one unit changes another unit’s outcome, whichever arm they are in. Colleagues talk, referrals happen, an auction reprices, a marketplace rebalances. The control arm moves because of what you did to the treated arm, and the direction depends on whether the interference is cooperative or competitive.

Intention to treat, and what it is promising

The standard defense is to analyze everyone in the arm they were assigned to, whatever happened afterwards. That is intention to treat, and it is not a compromise made for convenience.

Randomization makes the two arms comparable at the moment of assignment. Anything that happens after that, delivery failure, a rep intervening, a person ignoring the message, is downstream of treatment and often caused by it. Filtering on it destroys the comparability that the randomization created, which is why filtering an email test to people who opened manufactures lift out of selection.

Intention to treat also answers the question a budget owner is asking. They are not deciding whether to treat people who would definitely receive the message. They are deciding whether to run the program, delivery failures included.

Where it stops working

Intention to treat compares assigned-treated against assigned-control. It is unbiased for the effect of assignment as long as the control arm is what it would have been without the experiment.

Spillover breaks that. If treating half the market changed prices for the other half, or if treated users referred untreated ones, the control arm is no longer a picture of the untreated world. Intention to treat then faithfully and unbiasedly estimates the difference between two contaminated states, and no analysis of the data can recover the quantity you wanted.

The remedy is not statistical. It is the unit: randomize at a level that contains the interference, which is the design question asked before the test rather than after it.

Recovering the effect on the treated, and what it costs Technical

Where the leak is noncompliance and treatment is genuinely unavailable to the control arm, the effect among those treated is recoverable without conditioning on anything. Divide the intention-to-treat estimate by the difference in treatment rates between arms:

ITT   = Y_assigned_treated − Y_assigned_control
CACE  = ITT / (treated_rate_T − treated_rate_C)

This is the Wald estimator, and it rests on two conditions. Assignment must affect the outcome only through treatment, which fails whenever being assigned does something on its own, such as a rep noticing the account is in a program. And nobody may be a defier, taking the treatment when assigned to control and refusing it when assigned to treatment, which in a marketing setting is close to automatic since control units usually have a treatment rate of zero.

Note what it does not do. It rescales a diluted estimate, so it inherits the intention-to-treat interval scaled by the same factor, and a low treatment rate produces a large scaling and a very wide interval. It is a correction for dilution, not a way to get precision the design did not buy.

For spillover the two-stage design is the instrument: randomize clusters to different treated shares, then randomize units within them. Comparing untreated units across clusters with different shares estimates the spillover directly, and the total effect becomes the sum of a direct and an indirect part rather than a single number that quietly contains both.

Context

Best suited for

Any experiment where delivery is imperfect, where the control group can be reached by accident, or where the units talk to each other.

Data requirements

Assignment recorded separately from delivery, so intention to treat is computable at all
Delivery outcomes per unit, to measure the treatment rate in each arm
A record of control-arm contacts, since contamination that nobody logs cannot be quantified
Cluster membership where spillover is plausible, to test for a gradient across exposure

What changes by context

Lifecycle-led: Noncompliance dominates, because deliverability is the main leak and it correlates with engagement. Contamination is rare and usually operational, a service message sent to everybody including the holdout.
Product-led: Spillover within a workspace is the main risk, since colleagues see each other’s experience directly. That usually forces the workspace as the unit rather than the user.
Marketplace: Spillover is structural rather than incidental: treating demand changes what supply sees, so the control side is affected by construction. Market-level randomization is often the only design that estimates anything interpretable.
B2C ecommerce: Geo tests leak across market borders through travel and branded search, so estimates are lower bounds. Choosing separated markets reduces it and does not remove it.

When it breaks down

Experiments with no delivery record, where intention to treat cannot be distinguished from as-treated and the analysis has already conditioned on something post-treatment.
Treatment rates so low that the scaled estimate has an interval spanning everything, where the report is the intention-to-treat number alone.
Interference structures with no cluster boundary, such as a national brand campaign, where no unit contains the spillover and the design cannot be repaired.

What this does not establish

That intention to treat is conservative in general. It is conservative under dilution and can overstate under competitive spillover, which is the case people assume it protects them from.
That the scaled estimate is the effect on everybody. It is the effect among units whose treatment status was changed by assignment, which is a specific subgroup and usually not the whole population.
That a spillover gradient proves interference. A gradient across clusters can also reflect that high-treatment clusters differ, unless the treated share was itself randomized.
That measuring the leak repairs it. Quantifying dilution lets you rescale; quantifying spillover tells you the estimate is not what you wanted and does not produce the one you did.