Choosing Between User, Geo, and Cluster Holdouts

The design discussion usually opens with power. Person-level randomization gives the most units, so it gives the tightest interval, so that is the design.

Six weeks later the readout is clean and wrong, because the two people sharing the account were in different arms, or because the treated users told the control users, or because the ad could never have been withheld from an individual in the first place and the assignment was fiction.

Power is the easiest thing to compute and the last thing that should decide the unit.

Four questions, in this order

1. At what level is the treatment delivered? You cannot randomize finer than you can deliver. A broadcast, a billboard, or most connected TV inventory cannot be withheld from a person, so the person is not available as a unit whatever the power calculation says.

2. At what level does interference happen? If treating one unit changes another unit’s outcome, the two are not independent and the comparison is contaminated. Randomize at a level that contains the interference: the household if people share an account, the account if colleagues talk, the market if the auction price moves.

3. What can operations withhold, for how long? A design that requires a sales team to ignore named accounts, or a merchandiser to leave a market dark through peak, is not a design. It is a proposal that will be declined, and knowing that before the power calculation saves a week.

4. Only now, what can it detect? With the unit fixed by the first three, compute the detectable effect and decide whether the test is worth running. Often it is not, and that is a finding worth having in advance.

What each unit costs and what it protects against
Unit	Use it when	What it costs
User	Delivery is addressable and users do not influence each other	Nothing, which is why it is over-chosen
Household or account	People share a subscription, a device, or a purchase decision	Effective sample falls by the design effect, often by half or more
Cluster or team	Colleagues discuss the product or the treatment spreads socially	The same, and clusters are usually fewer and more unequal
Geo or market	Delivery cannot be targeted finer, or interference is local	Enormous. Sixteen markets is sixteen units, whoever lives in them
Time, switching back	Units are too few to split and the treatment can be toggled	Assumes this period does not carry into the next, which media violates

Reading down the table is reading in order of how much interference the design can absorb, and in order of how much precision it gives up to do so.

Clustering costs more than people expect

Randomizing households rather than people does not simply reduce the count from people to households. It reduces the effective count by how similar people inside a household are, and for purchasing behavior they are very similar.

That similarity is what a design effect measures, and it is the reason a clustered test with the same number of people as an individual one can have a fraction of the power. Computing it before the test, rather than discovering it in a wide interval afterwards, is most of the value of thinking about the unit at all.

Once the unit is chosen, the temptation is to recover the lost precision with a cleverer estimator. Where precision comes from runs the usual candidates on a geo test and finds that the interval is bought with units, not with estimators.

The design effect, and what it does to a sample size Technical

For clusters of average size m with intra-cluster correlation ρ, the variance of a clustered estimate is inflated by 1 + (m − 1)ρ relative to randomizing individuals. The effective sample size is the number of people divided by that factor.

The numbers are unforgiving. Households of 2.5 people with an intra-cluster correlation of 0.4 give a design effect of 1.6, so 100,000 people behave like 62,500. Teams of ten with a correlation of 0.3 give a design effect of 3.7, so the same 100,000 people behave like 27,000. And because precision follows the square root, cutting the effective sample by four widens the interval by two.

Two practical consequences. Estimate ρ from historical data before designing, because it varies enormously by outcome and guessing it wrong changes the answer by a factor. And where clusters are few and unequal, the effective count falls further still for the reason set out in the systems section: unequal units are worth less than their count, and a handful of large clusters can dominate a comparison regardless of how many small ones surround them.

Context

Best suited for

Any team about to run a holdout, and particularly one where the first proposal was person-level because it had the most units.

Data requirements

A stated delivery mechanism, so the finest available unit is known rather than assumed
Household, account, or cluster identifiers where interference is plausible
Historical outcome data to estimate the intra-cluster correlation rather than guess it
An operational commitment about what can be withheld and for how long, obtained before the design is finalized

What changes by context

B2C ecommerce: Person-level usually holds, with households the main exception and geo the fallback for upper-funnel media. Branded search across market borders is the most common leak.
Lifecycle-led: Delivery is addressable so the person is available, and the real constraint is contact capacity rather than the unit. Shared subscriptions are the case that forces a household design.
Product-led: Teams inside a workspace see each other’s experience, so the workspace is usually the unit whatever the signup grain suggests. Design effects here are large because teams behave coherently.
Marketplace: Interference is the defining feature: treating demand changes what supply sees. The market is often the only usable unit, and even then two sides tested at once contaminate each other.

When it breaks down

Treatments that cannot be withheld at any unit, such as a price change published everywhere, where the question moves to a modeled counterfactual.
Populations small enough that the coarsest safe unit leaves too few units to detect anything, where the right output is that the test should not be run.
Interference structures nobody can characterize, where the choice of unit is a guess and should be reported as one.

What this does not establish

That a coarser unit is always safer. It absorbs interference and gives up precision, and a geo test too underpowered to detect a plausible effect is not a more conservative choice, it is a more expensive way to learn nothing.
That an intra-cluster correlation measured on one outcome applies to another. It varies by outcome, by segment, and over time.
That containing interference removes bias. It removes interference between arms; confounding, attrition, and noncompliance are separate problems with separate remedies.
Any particular design effect for a given business. The numbers here are illustrative arithmetic, and the correlation has to be estimated on your own data.