Units Above the Visitor

The pricing test randomized users. Two people from the same company, one in each arm, compared screenshots in a shared channel and asked support why one of them was being charged 20% more. The test was stopped on day three, and the outcome that stopped it never appeared in any metric.

The visitor was the wrong unit, and the readout was never going to say so.

The decision it encodes

At what level is this surface randomized, and what does moving up cost?

The previous component assumes a visitor can be assigned an arm and the assignment stays with them. This one asks whether the visitor is the right thing to assign, which is a question about who else the treatment reaches.

The cases where the visitor is wrong

People who share a decision. In a product bought by a team, a price or a plan shown to one person is shown to the company. Randomizing seats puts colleagues in different arms, which is a support incident and a measurement failure at once: the outcome is the account’s, and an account with members in both arms belongs to neither.

Two sides of a marketplace. A change to the seller onboarding flow that brings more sellers changes what buyers see, and a buyer-side control arm is no longer a picture of the world without the change. The choosing a holdout design lesson calls this interference, and marketplaces have it by construction.

Referral and network loops. A signup flow with a referral step treats the people it refers, who may be in the control arm. The bigger the referral effect, the more the control arm has been treated, and the smaller the measured gap.

Shared devices and households. A streaming or grocery product used by a household is assigned per device and consumed per household. The unit that receives the treatment and the unit that produces the outcome are different sizes.

In every case the fix is the same shape: randomize at the level that contains the spillover. The account rather than the seat, the market or the category rather than the buyer, the household rather than the device. The cost is always the same too.

What moving up costs

Randomizing three thousand accounts of eight seats each is not a test on twenty-four thousand people. Seats inside an account behave alike, and the comparison is powered by the variation across accounts. With a within-account correlation of 0.5, the design effect is 1 + (8 − 1) × 0.5 = 4.5, and the 24,000 seats are worth about 5,333 independent ones. That is closer to the account count than to the seat count, which is the usual result: once units inside a cluster are correlated at all, the cluster count is what you have.

Detecting interference you did not design for

Some spillover is obvious in advance. Some is not, and a program that never checks will not find it. The standard check is a two-level design run occasionally rather than always: randomize clusters to a high or low share of treated visitors, then randomize visitors within each cluster. If the effect on treated visitors differs between high-share and low-share clusters, the treatment is leaking, and the size of the difference is the size of the leak.

It costs power, since it splits the sample twice, so it is a diagnostic to run on a surface once a year rather than a design to run on every test. The when the experiment leaks lesson covers what the result means and what to do about it.

Maturity, honestly staged

Stage 1: the unit named per surface. Signup at the visitor, pricing at the account, seller onboarding at the market. Written down, so the flag tool is configured to match rather than left at its default.

Stage 2: clustered readouts wherever the unit is above the visitor. Intervals stop being too narrow by a known factor.

Stage 3: the design effect in the power calculation. A test on accounts is planned as a test on accounts, so it is given the runtime it needs rather than stopped early on an interval that was never real.

Stage 4: a scheduled two-level diagnostic on the surfaces that matter. Interference is measured once a year rather than assumed away forever.

Design effect, clustering, and what the two-level design estimates Technical

For clusters of size m with intra-cluster correlation ρ, the variance of a cluster-randomized estimate is inflated by 1 + (m − 1)ρ relative to individual randomization, and the effective sample is the individual-level observation count divided by that. The division applies once. An outcome measured once per account, such as whether the account converted, already has the account as its observation, and dividing the account count by a seat-level design effect penalizes the test twice. With unequal cluster sizes the Kish adjustment applies on top, and a product where a few large accounts hold most seats loses more than the formula with average m suggests.

Analysis matches the randomization. Cluster-robust standard errors at the assignment level are the minimum; with few clusters, as in a B2B product with a few hundred accounts, they run anti-conservative and a wild cluster bootstrap or randomization inference over the cluster assignment is the safer interval.

The two-level design estimates a direct effect (treated against control within a cluster) and a spillover (control visitors in high-share clusters against control visitors in low-share clusters). The total effect of shipping to everyone is the sum, and it is the quantity a shipping decision needs; a visitor-level test estimates the direct effect alone and calls it the total.

Context

Best suited for

Products where any surface is seen by groups who share a decision, a market, or a device, which covers most B2B products, every marketplace, and consumer products used by households.

Data requirements

• A mapping from visitor to the containing unit: account, household, or market
• Cluster sizes, so the design effect can be computed rather than assumed
• A flag tool that can randomize on an identifier other than the visitor
• For the two-level diagnostic, the ability to set a per-cluster treatment share

What changes by context

B2B SaaS

The account is the unit for anything touching price, plan, or seats, and the sample is the account count. A few hundred accounts means a few tests a year with real power, which is closer to the sales-led system than to a consumer growth program.

Marketplace

Supply-side surfaces change what the demand side sees, so the market or the category is the unit and the design starts to look like a geo test. Switchbacks, alternating the treatment over time within a market, are the standard answer when there are too few markets to split.

Product-led

Referral and invite steps are the interference to watch. A signup flow that invites teammates treats the control arm through the people it recruits.

B2C subscription

Household plans mean the device is not the person and the person is not the payer. Randomize the account, read the account, and expect the effective sample to be the household count.

When it breaks down

• Products with a handful of large accounts, where randomizing at the account leaves nothing to compare and the design has to be staged rollout or a time-based switchback.
• Interference that does not respect any observable cluster, such as a public referral link shared on social media, where no level contains the spillover and the effect can only be bounded.
• Surfaces where the sharing structure is unknown and unrecorded, which is the case that produces confident wrong readouts rather than no readout.