Geo Design and Market Matching

The test ran in ten markets against ten controls the media team picked as comparable. Six weeks later the treated markets are 3% ahead, the deck says the channel works, and the budget moves.

Then somebody plots the year before. The treated markets were already pulling away from the controls, at about the same rate, months before anything changed. The estimate was a trend nobody checked, and the design had no way to notice, because the markets were chosen for how similar they looked rather than for how similarly they had behaved.

The decision it encodes

Which markets change, for how long, and what can that detect?

The third clause has to be answered before the first two are fixed, and usually is not. A design chosen without a detectable effect attached is a design that will produce a number regardless of whether anything happened.

Four quantities, written down before launch

A geo test that reports one number has usually blurred four. Each is a different quantity, and the readout carries all of them.

What every test pre-specifies
Quantity	The question it answers
Assignment effect	What did assigning a market to the spend policy do to the outcome? The intention-to-treat estimate, and the headline.
Delivered contrast	How much did spend, impressions, and reach actually differ between arms? A blackout that leaked or a scale-up that never delivered changes what the headline means.
Business effect	The incremental outcome in the units the budget is judged on: orders, activated accounts, contribution, over the stated horizon.
Efficiency	Incremental outcome or contribution per incremental dollar, from the business effect and the delivered contrast. The number the allocation consumes.

The first is what randomization protects. The last is what the budget wants. The two in between are how you get from one to the other without pretending the spend moved exactly as planned.

Your sample size is markets, not people

This is the fact that surprises people who arrive from person-level testing. A geo design randomizes markets. Two hundred million customers spread across twenty markets gives twenty units, and the precision of the estimate follows that twenty.

It follows the residual variance between markets too, which is what matching is for. Matching does not make markets comparable in some absolute sense. It reduces the variance of the difference, and that reduction is the entire reason a design with twenty units can detect anything at all.

Going dark is not the only design

Turning spend off in a set of markets is the cleanest version and the one most often refused, because it asks the business to forgo revenue in named places for a named period.

Scaling up is the same design with the sign reversed, and it is usually easier to authorize. It answers a slightly different question, the return on the increment rather than the return on the base, and that is frequently the question worth asking anyway, since the decision on the table is where the next dollar goes rather than whether to keep the channel at all.

Switchbacks alternate treatment and control in the same markets over time, which recovers units when there are too few markets to split. They buy that back by assuming this week’s spend does not carry into next week, which for media with any adstock at all is the assumption you cannot make casually.

Randomize within the matched pairs

Having found pairs, the assignment inside each pair should be a coin flip rather than a judgment. Hand-picking which of a matched pair goes dark reintroduces the selection the matching removed, usually through a preference for protecting the market somebody cares about.

The registry pattern from person-level work applies unchanged: record the pairs, the assignment, the salt, the dates, and the spend that was supposed to move, so that the readout can check the treatment happened before it estimates the effect of it.

A leaking holdout fails quietly

A holdout is a promise that a set of markets did not see the campaign, and almost nobody verifies the promise. A campaign duplicated during a creative refresh does not inherit the geo exclusions. An automated bidding strategy expands targeting to hit its volume goal. Someone launches mid-flight with last quarter’s location list. In each case the held-out markets receive advertising, the measured gap shrinks toward zero, and the test returns a small, plausible, well-behaved estimate biased toward the conclusion that the channel does nothing. It does not return an error.

The fix is ten minutes a week: pull delivery by market for the whole window and confirm the contrast the design promised. For a blackout that means held-out markets received zero impressions, since any delivery there is a leak. For a scale-up it means the spend difference between arms is the one the design specified. Either way the achieved contrast and any leakage are reported with the estimate, because the estimate is about the contrast that happened, not the one that was planned. Check in week one, because a leak caught then costs a restart and a leak caught at analysis costs the test. And log the check, because the estimate will be unwelcome to somebody six months later and the delivery audit is what makes it defensible.

Contamination is a property of the medium

Markets leak. People commute, streaming platforms deliver against a household that spans a border, and brand search rises in a dark market because somebody saw the ad on holiday. The estimate is then a comparison between treated and partly-treated, which understates the effect.

The practical response is to choose markets that are geographically separated rather than adjacent, to check whether the channel can even be bought at the grain you are randomizing, and to say plainly in the readout that the estimate is a lower bound where leakage is likely.

Maturity, honestly staged

Stage 1: matched controls and a stated detectable effect before launch. Most of the value. A design that cannot detect the effect under discussion should be changed or abandoned while that is still cheap.

Stage 2: randomized assignment within matched pairs. The selection the matching removed stops coming back in through the choice of which market to darken.

Stage 3: pre-trend and placebo tests as launch gates. Divergence is caught before six weeks are spent, rather than in the meeting where the result is presented.

Stage 4: a standing library of pairs with known residual variance. Power calculations stop being bespoke, because the variance that drives them has been measured rather than assumed, and the next test can be scoped in an afternoon.

Where the detectable effect comes from Technical

The estimator is a difference in differences: the change in the treated markets, less the change in the matched controls. With n₁ treated markets, n₀ controls, and σ the standard deviation of a market’s period-over-period change after matching, the smallest effect a two-sided test at conventional levels can detect is roughly

MDE = (z_1−α/2 + z_1−β) · σ · √(1/n₁ + 1/n₀)

Two things follow. The number of markets sits under a square root, so doubling them improves the detectable effect by about 29%, which is why twelve markets and twenty markets are not the different worlds people expect. And σ is the term matching acts on, so a good pairing buys more than adding markets usually can.

The formula is the first approximation and should not be the plan. Market outcomes are unequal and heavy-tailed, consecutive weeks are not independent draws, and a pair estimator’s precision depends on how well the pairs actually track. The planning method that sees all of that is simulation: take the markets’ own history, draw many placebo assignments with the planned pairing, run the planned estimator with a fake launch date, and read the spread of estimates that come back when nothing happened. That is the test’s real noise floor, and where precision comes from shows how far it can sit from the formula. It is also why there is no universal minimum market count: ten large, well-paired markets can outperform thirty small noisy ones, and the answer for a given program comes from its own markets.

Synthetic control generalizes the matching: instead of picking control markets, it fits a weighted combination of untreated markets that reproduces the treated market’s pre-period, and uses that combination as the counterfactual. It is stronger where one treated market is large and unusual, and it needs a long pre-period to fit the weights on. Both approaches rest on the same assumption, which is that the pre-period relationship would have continued, and neither can test it directly for the period that matters.

The geo holdout designer works these numbers for a given market count and baseline, and

designing a credible geo holdout

runs one end to end, including the diagnostics that decide whether it can be believed.

Context

Best suited for

Channels that cannot be withheld from an individual, where the finest thing you can turn off is a market, and where enough comparable markets exist to make a difference readable.

Data requirements

Outcome by market and week, for long enough before the test to match on and to check trends against
Spend by market and day, so the readout can confirm the treatment landed where it was assigned
A market definition the media platforms can target, rather than one only the warehouse recognizes
A record of promotions, launches, and distribution changes by market, since those are the shocks that break the design

What changes by context

B2C ecommerce: Markets are cheap to define and fast to read, so this is the natural home for geo work. Watch branded search across borders, which is the most common contamination path and the one that makes an effect look smaller than it is.
B2C subscription: The outcome accrues for months, so a six-week window measures signups rather than retained value. Either run long enough to see the real outcome or state plainly that the estimate is on a proxy and say which one.
Local / omnichannel: The strongest setting for this design, because trade areas are real rather than administrative and offline outcomes are already reported by location. Store openings and closures are the shock to guard against.
Marketplace: Supply and demand interact within a market, so darkening demand media changes the experience of sellers too. The unit is the market for a reason, and the outcome should be a market-level one rather than a per-user average.

When it breaks down

Too few markets for the achieved contrast, the pairing, and the variance to detect an effect worth acting on, which a placebo simulation on the actual markets decides. There is no universal count, and a handful of markets is usually below it.
Media that cannot be bought or withheld at the geo grain, including much of connected TV and most partner inventory.
National launches, price changes, or promotions that coincide with the window, which move both arms and cannot be separated after the fact.
Markets so interconnected that leakage dominates, where the estimate is a lower bound of unknown looseness rather than a measurement.