Where Precision Comes From in a Holdout
Pairing, covariate adjustment, a weighted control, and randomization inference, applied to one geo test in two noise worlds. Most of the levers people reach for buy nothing in a geo test. The design and the inference are what move the interval.
The geo holdout example ends with an interval about seven points wide, and the reflex of everyone who reads it is to ask how to narrow it without asking operations for more markets. The methods for doing that have names: analyze the pairs, adjust for the pre-period, build a weighted control from the donor markets, run randomization inference. Each is a real tool. What is less often said is that they do different amounts of work in different places, and that in a geo test most of them do almost none.
This lesson runs all four on the same design, in two versions of the world, and reports what each one bought.
The same test in two kinds of noise
Take the holdout design: 40 markets, paired on baseline volume, one of each pair treated by a coin flip, 8 weeks before the test and 6 during it, a true lift of +7.5%. The estimators are then handed two datasets.
In the quiet world, each market-week has its own independent noise and nothing else. This is the world a textbook power calculation assumes and the one in which every conventional standard error is exactly right.
In the real world, the noise has structure. Each market carries a shock that persists from week to week, decaying slowly rather than resetting. Markets in the same region move together, so a good week in one is a good week in its neighbors. The independent per-week noise is smaller to compensate, so the total variance is comparable. This is what a regional panel of weekly conversions looks like, and it is where a conventional standard error goes wrong.
The five reads are the same in both worlds. Conventional is the two-way fixed-effects regression with an ordinary standard error, treating every market-week as an observation. Analyze the pairs collapses each pair to one number and runs a t test on twenty of them. Adjust for the pre-period adds each market’s pre-period level as a covariate, the geo version of CUPED. Weighted control replaces the plain control average with a weighted combination of control markets chosen to match the treated group’s pre-period, the synthetic-control idea applied inside a randomized test. Randomization inference re-flips the coin in every pair fifteen hundred times and reads the interval off the distribution of estimates that would have come out under no effect.
Show the data
| Estimated lift (%) | Point | 95% CI |
|---|---|---|
| Conventional | +6.5% | +3.2% to +10.0% |
| Analyze the pairs | +6.5% | +2.9% to +10.3% |
| Adjust for the pre-period | +6.5% | +3.1% to +10.1% |
| Weighted control | +8.4% | +3.5% to +13.6% |
| Randomization inference | +6.5% | +2.2% to +11.1% |
Show the data
| Estimated lift (%) | Point | 95% CI |
|---|---|---|
| Conventional | +8.2% | +2.5% to +14.3% |
| Analyze the pairs | +8.2% | +0.5% to +16.6% |
| Adjust for the pre-period | +8.2% | +1.3% to +15.6% |
| Weighted control | +10.3% | -2.9% to +25.2% |
| Randomization inference | +8.2% | +0.2% to +16.8% |
What each lever bought
Nothing narrowed the interval. That is the result, and it is worth sitting with before reaching for a reason. In the quiet world the pairs, the covariate adjustment, and the re-randomization all reproduce the conventional width because the conventional width is correct there. In the real world the three principled reads agree with each other and disagree with the regression, and the disagreement runs the wrong way for anyone hoping for a narrower answer.
Pairing changed the standard error, not the estimate. With a balanced panel and one treated market per pair, the regression coefficient equals the average of the twenty pair differences exactly. The point estimates match to the decimal. What pairing buys is the right count of observations: twenty pairs, not 560 market-weeks. In the real world that is the difference between an interval 11.8 points wide and one 16.1 points wide. The narrower one is wrong.
Adjusting for the pre-period did nothing, for a specific reason. CUPED and its regression cousins earn their keep when the covariate predicts the outcome and the design has not already used it. In a difference-in-differences on log outcomes the pre-period is already subtracted. The fitted coefficient on the pre-period level was 0.99 in the real world and 1.00 in the quiet one, which is to say the regression put the covariate in with a weight of one and recovered the differencing it already had. There was no residual variance left for it to explain.
The weighted control was wider in both worlds, and biased in one. With eight weeks of history and twenty donor markets, a simplex weight vector has more freedom than the data can discipline, so it fits the pre-period noise and carries that fit into the test window. In the quiet world there was nothing structured to match, so the fit was pure overfitting and the interval came out 1.48×. In the real world there was regional structure to match, and the weights still lost to the plain average, because twenty randomly assigned treated markets already average the regional shocks out. The plain control mean of twenty markets is itself a very good synthetic control, and it has no parameters.
Giving the weights a year of history instead of eight weeks helps some. With 52 pre-period weeks the weighted read in the real world came in at 0.88× the width of the paired read, a modest gain rather than a rescue. That is the scale of what the method has to offer inside a randomized design of this size: a few percent when the history is long and the structure is real, and a cost when either is missing.
- Treated markets
- Plain control average
- Weighted control
Show the data
| Week relative to launch | Treated markets | Plain control average | Weighted control |
|---|---|---|---|
| W-8 | -5.6 | -2.9 | -4.0 |
| W-7 | 6.1 | 5.7 | 5.9 |
| W-6 | 5.7 | 4.1 | 4.9 |
| W-5 | 8.0 | 6.1 | 8.8 |
| W-4 | 0.2 | 3.5 | 1.8 |
| W-3 | -1.8 | -4.4 | -1.2 |
| W-2 | -10.8 | -9.5 | -14.3 |
| W-1 | -0.4 | -1.3 | 0.1 |
| W1 | 5.8 | 1.6 | 4.4 |
| W2 | 7.0 | -1.3 | 3.9 |
| W3 | 16.9 | 9.2 | -1.2 |
| W4 | 18.2 | 9.8 | 8.1 |
| W5 | 19.0 | 8.9 | 1.4 |
| W6 | 6.1 | -6.1 | -6.4 |
Where the interval actually moves
If estimators do not narrow a geo interval, the design does, and the two design levers are not equal. Doubling the markets and doubling the weeks both double the observations in the regression’s count. Under persistent noise they buy very different things.
| Design | Markets | Test weeks | Interval width | Relative |
|---|---|---|---|---|
| Base design | 40 | 6 | 13.0 points | 1.00× |
| Twice the markets | 80 | 6 | 8.4 points | 0.65× |
| Twice the weeks | 40 | 12 | 11.9 points | 0.92× |
Twice the markets narrowed the interval to 0.65×, close to the 0.71× that independent units promise. Twice the weeks narrowed it to 0.92×. Consecutive weeks of the same market are not independent draws when its shock persists from one to the next, so the second six weeks mostly re-observe the noise the first six already recorded. This is why a geo test that ran long and still came back wide is not a puzzle, and why the honest answer to “can we run it longer instead” is usually that it will not help much. A power calculation that counts weeks as independent will promise a width the test cannot deliver.
Where the levers do work
None of this argues against variance reduction. It argues for knowing where the variance is.
At the visitor level the arithmetic is reversed. A visitor’s pre-period behavior predicts their outcome, the randomization has not used that information, and there are tens of thousands of units, so the covariate has a great deal of residual variance to explain and the weight of one unit’s noise is negligible. Adjusting the same 40,000-visitor test for a pre-period covariate correlated 0.55 with the outcome narrowed the interval to 0.84×, exactly the √(1 − ρ²) = 0.84 that the formula promises. That is a free 16 percent, and it is why the growth experiments system treats regression adjustment as a default rather than a technique.
The pattern generalizes. Covariate adjustment buys precision in proportion to how much the covariate predicts that the design has not already removed. Weighted controls buy it when the treated side is one or a few units and the donors have structure worth matching, which is the observational case rather than the randomized one. Pairing and randomization inference do not buy precision at all; they buy a correct account of the precision the design already has, which in a panel with persistent noise is less than the regression reports. For twenty pairs of markets, that correction is the most important thing the analysis does.
How the two worlds and the five reads were built Code
40 markets with log-normal baselines, paired on baseline and assigned by one coin per pair, 8 pre-period and 6 test weeks, a shared seasonal curve, and a true lift of +7.5% on treated market-weeks. The quiet world adds independent log-normal noise with standard deviation 0.07 per market-week. The real world adds an AR(1) shock per market with innovation 0.05 and persistence 0.85, two regional factors with weekly standard deviation 0.08 loaded by each market, and independent noise of 0.03.
Conventional is two-way fixed effects on log conversions with an ordinary standard error. Pairs is a t test on the twenty pair-level difference-in-differences with nineteen degrees of freedom. Adjusted adds each market’s pre-period mean to the pair-level regression. The weighted control uses Frank-Wolfe on the simplex, sixty iterations so weights stay spread, fitted to the treated group’s pre-period mean; its interval comes from re-flipping the pairs and refitting the weights each time. Randomization inference re-flips one coin per pair 1,500 times. The design table simulates each design 200 times and reports 2.8 times the spread of the paired estimate. The visitor block draws 40,000 visitors per arm with a pre-period covariate correlated 0.55 with the outcome and compares the difference in means with the regression-adjusted difference.
Context
Best suited for
Anyone who has been asked to make a geo interval narrower without more markets, and anyone about to promise that a longer test will do it.
Data requirements
- The assignment log with the pair structure and the coin flip for each pair, so pairs can be analyzed and re-randomized
- Enough pre-period history to check whether shocks persist, which decides whether weeks can be counted as independent
- Market-level covariates that were not used in the pairing, if adjustment is to be tried at all
- For visitor tests, a pre-exposure metric per unit, which is what makes adjustment worth the trouble
What changes by context
- B2C ecommerce
- Visitor tests are where adjustment pays, and the pre-period covariate is usually trailing purchase rate. Geo tests for upper-funnel media get their precision from market count and should be designed that way.
- Local / omnichannel
- Store markets carry persistent local shocks, so the real-world column is the relevant one and a long test buys less than it seems. Adding markets, even small ones, is the lever.
- Product-led
- Workspace-level tests sit between the two cases: hundreds of units rather than twenty, and a strong pre-period covariate in prior usage, so adjustment recovers a real fraction of the width and clustered inference remains necessary.
When it breaks down
- Designs with very few treated markets, where the plain control average is no longer a good synthetic control and weighting with a long history becomes worth its overfitting risk.
- Outcomes with a strong market-level covariate that pairing did not use, such as a prior test in the same markets, where adjustment can recover real width.
- Tests where the assignment record is incomplete, since every read here except the conventional one depends on knowing what the coin did.
What this does not establish
- That weighted controls are useless. They are the right tool when nothing was randomized and the treated side is small. Inside a randomized test with twenty treated markets they have little to do.
- That covariate adjustment never helps a geo test. It helps when a covariate predicts the outcome beyond what the pre-period difference already removed. Here nothing did, and that is the common case rather than the universal one.
- That the width ratios transfer. A conventional standard error that understates by 1.4 is a property of how persistent this simulation made its shocks. The direction is general; the size depends on the panel.
- That a longer test is never worth running. It is worth running when the effect itself needs time to appear, which is a different reason from precision and the right one to give.