Designing a Credible Geo Holdout

Paid social prospecting is running at roughly $47,500 per week. The channel reports a solid return on last-touch. Finance has asked whether the budget should grow, hold, or move to search.

The decision is not whether paid social works. It is whether the next dollar here buys more than the same dollar somewhere else. That needs the number of conversions this spend creates, not the number it gets credited with.

The channel reports 4,820 last-touch conversions for the quarter against $285,000 of spend. That is a reported CAC of about $59, comfortably inside target. On this number, the answer is obviously to spend more.

Reported CAC counts conversions the platform could claim. That includes people who saw an ad and would have bought anyway, people already mid-journey, and people who converted through another route but had an impression in the lookback window.

None of that is dishonest reporting. Credit is not causation, and the size of the gap stays unknown until you measure it. Prospecting reaches colder audiences, so its gap is usually smaller than retargeting or branded search. “Usually smaller” is not a number you can put in a budget model.

Available. Conversions attributable to a market by billing postcode, daily, back three years. Spend controllable at the market level in the ad platform. Clean market definitions that match how the media is bought.

Missing. Any user-level link between ad exposure and purchase. Consent rates make that unreliable, which is what pushes this toward a geo design rather than a user-level one. Also missing: any prior experiment on this channel, so there is no existing estimate to anchor on.

The estimand is the average effect of running paid social prospecting on conversions in a market, during the weeks it runs, for markets like these.

That phrasing carries three limits, and each one matters later:

• During the weeks it runs. Not a persistent effect, not a brand effect that builds over a year.
• At this spend level. Effects are not linear in spend, so this says nothing about doubling.
• For markets like these. Forty markets chosen for testability, not a random sample of geography.

40 markets, split 19 treated and 21 held out, with an 8-week pre-period and a 6-week test.

Markets are assigned in matched pairs: rank by baseline conversion volume, then randomize within each adjacent pair. Pure random assignment across markets that differ by an order of magnitude leaves too much to chance, since one large market landing in a single arm can dominate the comparison. Pairing forces balance on the variable most likely to matter. Randomizing inside the pair keeps the randomization that makes the comparison credible.

The resulting arms are balanced on baseline volume to within 17.9%.

Was it powered?

Before running, the question is what lift this design could detect at all. Using the observed within-market week-to-week variability (σ = 0.095 on the log scale) and 20 markets per arm:

Six weeks reaches an MDE below the roughly 5% lift that would change the budget decision, at a cost of about $285,000 in held-out spend. Going longer buys precision slowly: standard error falls with the square root of duration, so doubling the test narrows the interval by only about 30%.

You can run this calculation for your own design with the geo holdout designer.

The pre-trend check

The diagnostic that decides whether the headline number deserves belief. Both arms are indexed to their own pre-period average, because the arms differ in absolute size and raw totals would only show two parallel bands.

This is a passing diagnostic, not a perfect one. The pre-period gap of +1.3% per week is small against the noise, but it is not zero, and eight pre-period weeks gives the check limited power to catch a slow divergence. It makes parallel trends plausible. It does not make it proven, and no amount of pre-period data would.

The estimator

Two-way fixed effects on log conversions:

log(conversions[market, week]) = α[market] + τ[week] + β · (treated × post) + ε

Market fixed effects absorb persistent size differences. Week fixed effects absorb the shared demand curve: the seasonality and category drift that a naive before-and-after comparison would misread as campaign effect. What remains in β is the differential change in treated markets during the test window.

Converting the percentage lift into conversions: treated markets recorded 48,383 conversions during the test. At an estimated lift of +7.2%, roughly 3,246 of those were incremental, which puts about 45,137 in the “would have happened anyway” column.

Against $285,000 of spend, that is an incremental CAC of about $88, against a reported CAC of $59.

The interval matters more than the point. At the optimistic end (+10.5%) incremental CAC is about $62. At the pessimistic end (+4.0%) it is about $154. Those two numbers support different decisions, and this test does not separate them.

Hold the budget. Do not grow it on this evidence. Re-test at a higher spend level before committing more.

The reasoning:

• The channel works. The interval clears zero by a comfortable margin, so this is not a channel to cut.
• Incremental CAC of about $88 runs well above the reported $59, but stays inside the range the business tolerates for new customers.
• Growing the budget requires the marginal return. This test measures the average effect at current spend. For a channel that may be nearing saturation, those two can differ a great deal.

The summary for the budget meeting: this channel creates value, reported CAC overstates it by roughly a third, and we do not yet know what the next dollar buys.

1. A spend-level test. Split treated markets into two spend tiers and estimate the response curve instead of a single point. That answers the question this test could not.
2. A longer post-window. Continue measuring held-out markets for six weeks after the campaign resumes, to separate genuine demand creation from pull-forward.
3. Branded search. Higher spend, better reported performance, and a much wider plausible range of true incrementality. Per dollar of held-out spend, it is almost certainly the better next test.
4. A persistent holdout. A permanent 5% held-out set of markets turns incrementality from a project into a standing number, and makes ratio calibration practical.