The Persistent Geo Holdout

The calendar has a slot for paid social in the third quarter. In the second quarter the channel’s creative is refreshed, the platform changes its bidding model, and a competitor enters two of the largest markets. By the time the test runs, the coefficient it was scheduled to check no longer describes the channel anyone is spending against.

Nobody did anything wrong. A calendar produces evidence a few times a year, and the channel moved between readings.

The decision it encodes

Which markets are dark on which channels right now, and when do they rotate?

The test calendar decides which question gets the next dedicated test. This component decides something narrower and more permanent: that on the one or two channels carrying most of the budget, a comparison exists every week whether or not anyone asked for one.

What a standing control changes

The lifecycle system’s persistent holdout is why that system closes weekly. Every campaign reads out against a slice of the list that was withheld from everything, so no campaign needs its own negotiation. Paid media has no equivalent by default. Each geo test is a study, designed and argued for on its own, and between studies the model runs on an anchor that ages.

A persistent geo holdout is the closest paid media gets. A small share of markets is held dark on a channel at all times, rotating on a schedule so no market stays dark for long. The comparison between dark and lit markets is always available, which means calibration has a recurring anchor, a material shift in the channel’s effectiveness produces low-precision evidence within a rotation or two rather than at the next scheduled test, and the appetite decision is made once rather than per quarter.

Why it rotates

A fixed dark set, drawn at random, remains a valid randomized control. What changes is the question it answers. Over time it estimates the effect of a long-running policy of never advertising there, as brand awareness decays, competitors move in, and the sales team learns which markets are dark and behaves differently. That is a legitimate quantity and it is not the one a budget decision needs, and the dark markets drift away from current operating conditions in ways that have nothing to do with the channel. Rotation trades that continuity for coverage: every market takes a turn, the comparison stays close to how the channel runs today, and the cost is spread so no region carries it alone.

Rotation also changes what is being measured, and the readout has to say so. A market going dark carries the last few weeks of the channel’s effect with it, because adstock does not stop when spend does. A market coming back carries none. The estimate from a rotating design is the effect of pausing the channel for one rotation, read after a washout, which is a different quantity from the steady-state contribution a permanent holdout would estimate and from the marginal return a budget decision needs. All three are legitimate. Confusing them is the failure the response curves component exists to prevent.

What a rotating holdout can detect, at 100 markets and a market-level residual of 8%
Share of markets dark	Dark markets	One rotation	Four rotations pooled
5%	5	10.3%	5.1%
10%	10	7.5%	3.7%
20%	20	5.6%	2.8%
A dedicated calendar test, half and half	50	4.5%	once

Detectable effects are on total sales in a market, at conventional power. One rotation of a small holdout sees almost nothing. Pooled across a year, a ten percent holdout approaches what a dedicated test detects, on the condition that the effect held still across the year, which is the assumption the design has to state and the calendar’s test does not.

What it is for, and what the calendar is still for

The table is the argument for using both. A standing holdout on the largest channel gives calibration a recurring, low-precision anchor that accumulates, and it produces evidence that a channel’s effect has moved before the model has been refitted on a year of spend that assumed it had not, though at the precision the table shows rather than the week it happens. The calendar gives a sharp, single answer to the question with the most money on it.

The two share the same market pool, so the holdout’s dark markets are removed from the calendar’s available capacity for as long as they are dark. A program with a hundred markets and a ten percent standing holdout schedules its dedicated tests on ninety, which is a cost worth writing down beside the other one.

Maturity, honestly staged

Stage 1: one channel, a fixed dark set, one quarter. A pilot rather than a system, and enough to learn whether the organization will tolerate dark markets at all, which is the harder question.

Stage 2: rotation on a schedule. Markets enter and leave the dark set on a published cadence, with a washout window before each readout, so the comparison stays a comparison.

Stage 3: the holdout feeds calibration directly. Each rotation’s estimate lands in the calibration step as an anchor with a date, and the model’s coefficient for that channel is checked against it every refit rather than at the next scheduled test.

Stage 4: drift alerts. Successive rotations are compared to each other, and a channel whose rotations disagree beyond their intervals is flagged and moved up the calendar’s queue.

Estimating from a rotating design Technical

Each rotation is a two-arm comparison between dark and lit markets over one period, estimated the same way the dedicated test is: difference in differences on log outcomes against each market’s own pre-period, with the interval from the market-level spread. Pooling rotations is a meta-analysis of those estimates, and the pooled interval is only as narrow as the stability assumption is true, which is why the rotations are reported individually first.

Because markets enter and leave treatment at different times, the pooled version is a staggered design, and the usual two-way fixed effects regression on the full panel is not the right estimator for it when effects differ across rotations. Comparing each rotation’s dark markets only to markets that were lit throughout that period avoids the problem, at the cost of a little precision. The staggered rollout lesson shows the size of the error the shortcut makes.

The washout matters more than it looks. A market leaving the dark set has had no spend for a rotation and will take weeks to return to its adstocked level, so it is not a clean member of the lit arm until it has. Excluding recently rotated markets from the comparison for a stated number of weeks is a design choice that has to be made before the readout, and recorded.

Context

Best suited for

Programs where one or two channels carry most of the media budget, a mix model is refitted on a cadence, and nobody can say when that channel's coefficient was last checked against a live comparison.

Data requirements

Spend and outcomes by market and week, from the spend spine, at a grain that can tell a dark market from a lit one
A market definition the platforms can exclude, since a market that cannot be made dark cannot be held out
A rotation schedule and a washout rule, written down before the first rotation reads out
The dedicated test calendar, so the two designs do not claim the same markets

What changes by context

B2C ecommerce: The natural home. Short purchase cycles mean a rotation can be as short as six to eight weeks, and the promotional calendar sets which weeks are usable rather than the design.
B2C subscription: Acquisition effects show quickly and retention effects do not, so a rotation readout on signups is available long before one on retained value. Report both horizons and say which the anchor uses.
Local / omnichannel: The strongest case, since markets are the natural unit and store-level outcomes are observed. The complication is that going dark in a market with stores can be a commercial decision for the region, which is why rotation is what makes it tolerable.
Marketplace: Supply and demand spend interfere across a market boundary, so a market dark on one side is not a clean control for the other. Hold out one side at a time and say so.

When it breaks down

Programs with fewer than roughly forty usable markets, where even a large dark share reads nothing in a rotation and pooling a year still falls short of a decision.
Channels that are a small share of revenue, where the effect on total sales in a market is below anything a market design can see. The calendar cannot help those either.
Media that cannot be excluded by geography, including most national broadcast and sponsorships.
Organizations that will not tolerate any market being dark on the largest channel, where the answer is a smaller channel first and the argument later.