When Randomization Is Impossible
Difference-in-differences, synthetic control, and interrupted time series. What each one assumes, the diagnostic that would falsify it, and how far wrong the same estimator goes when the assumption fails.
The rollout went region by region, in whatever order operations could manage. Or a partner launched in some territories and not others. Or a competitor withdrew from half your markets and you would like to know what that was worth.
Nobody randomized anything, and the question still has to be answered. The designs for this are well known. What is less often said out loud is that they are not better or worse than each other in general. Each is reliable when one specific assumption holds, and that assumption is about how treatment came to be assigned.
Two worlds, one difference
Below are 60 markets over 52 weeks, with a true effect of +6.0% applied to 15 of them for the final 12.
Everything about the two worlds is identical except how those 15 were picked. In one, the markets already growing fastest adopted first, as a rollout prioritized by recent performance does. In the other, they were picked at random.
The estimators do not know which world they are in.
- Markets that adopted first
- Markets that did not
Show the data
| Week relative to launch | Markets that adopted first | Markets that did not |
|---|---|---|
| -40 | -6.4 | 1.4 |
| -36 | -2.2 | 3.3 |
| -32 | -1.2 | 2.7 |
| -28 | -4.4 | -2.0 |
| -24 | -5.7 | -5.0 |
| -20 | -5.8 | -6.9 |
| -16 | 2.8 | 0.4 |
| -12 | 5.0 | 1.7 |
| -8 | 8.9 | 2.7 |
| -4 | 11.0 | 2.6 |
| 0 | 10.1 | -5.8 |
| 4 | 10.2 | -6.6 |
| 8 | 13.6 | -5.2 |
What each estimator returns
| Design | Chosen by trend | Bias | Assigned at random | Bias |
|---|---|---|---|---|
| Naive post-period comparison | +24.4% | +18.4pp±1.43 | +7.1% | +1.1pp±1.10 |
| Difference in differences | +16.3% | +10.3pp±0.19 | +6.2% | +0.2pp±0.25 |
| Synthetic control | +11.9% | +5.9pp±0.18 | +6.1% | +0.1pp±0.15 |
| Interrupted time series | +0.3% | −5.7pp±0.15 | +0.4% | −5.6pp±0.15 |
Difference in differences is nearly unbiased on the right and off by +10.3pp on the left. It is the same estimator on the same data generating process. What changed is who decided which markets got treated.
Three readings, in order of how often they get missed.
The estimator is not the thing that failed. Difference in differences returns +6.2% against a true +6.0% when assignment is random, and +16.3% when the fastest-growing markets went first. Nothing about the method changed. Choosing a fancier estimator does not address the reason it went wrong.
Interrupted time series is wrong in both worlds. It returns about +0.4% either way, because it extrapolates the treated markets’ own trend and never looks at a control group. Anything else happening during the post period, a season, a price change, a category-wide slowdown, lands inside the estimate. Randomizing who gets treated does not help a design that ignores the comparison.
Synthetic control degrades more gracefully and does not rescue you. Its bias under selection is +5.9pp against difference in differences at +10.3pp, because reweighting the donor markets absorbs some of the trend difference. It is still wrong by roughly the size of the effect it is trying to measure.
The diagnostic that tells you which world you are in
Run the design on a period where nothing happened. Split the pre-period in half and estimate the effect of a launch that did not occur.
The pre-trend slope comparison says the same thing more quietly: the treated markets grow 0.36 log points a week faster before launch in the selected world, against 0.00 when assignment is random.
What each design assumes, and what would falsify it
Difference in differences. Assumes parallel counterfactual trends. Falsified by a non-zero placebo estimate in the pre-period, or by diverging pre-trends. Cheap, transparent, and the most commonly misapplied of the three.
Synthetic control. Assumes a weighted combination of untreated units can reproduce the treated unit’s pre-period, and that the relationship holds afterwards. Falsified by a poor pre-period fit, and by running the same procedure on untreated units to see how often it finds effects where none exist. Best when you have one or a few treated units and many clean donors.
Interrupted time series. Assumes nothing else changed at the moment of interruption. Falsified by any concurrent event, so it needs a control series to be credible and stops being interrupted time series once it has one. Reasonable for abrupt, well-dated changes in stable environments, and dangerous for anything seasonal.
None of these is a weaker version of an experiment. They are answers to the question of what to do when the assignment mechanism was somebody else’s decision, and their credibility comes from understanding that decision rather than from the estimator.
Context
Best suited for
Effects you cannot randomize: staged rollouts, partner launches, competitor moves, regulatory changes, and anything already in flight before measurement was consulted.
Data requirements
- A pre-period long enough to test the assumption, which usually means at least twice the post-period
- Untreated units that plausibly share whatever drives the treated ones, not units that happen to be available
- Knowledge of how treatment was assigned, which is the input the estimator cannot supply
- Outcome measured identically across units, since a definition change between regions looks like an effect
What changes by context
- Local / omnichannel
- The natural home for these designs, since stores and regions give many units and rollouts are staggered. The risk is that the rollout order was chosen by expected performance, which is the failure case above.
- Partner-led
- Partners decide their own launch timing, so assignment is not yours to control or even to observe. Ask the partner why they launched where they did; that answer is the identification argument.
- B2C ecommerce
- Geography is often a weak unit because the market is national and shipping is uniform, so donor markets are less distinct than they look and shared shocks dominate. Time-based designs tend to be worse here for the same reason.
When it breaks down
- Treatment assigned on the basis of the outcome itself, such as rescuing markets that were declining. No pre-period check saves this.
- Spillover between treated and untreated units, which contaminates the control group and understates the effect.
- Too few untreated units to build a credible comparison, where synthetic control fits the pre-period by overfitting it.
- Short pre-periods, where the diagnostic has no power to detect the divergence it exists to find.
What this does not establish
- That synthetic control is better than difference in differences. It was less biased here because the selection was on trend, which reweighting partially absorbs. Selection on level, or on something unobserved, would not go the same way.
- That the biases transfer. Ten percentage points of bias is a property of how strongly assignment correlated with trend in this simulation. The direction is general and the size is not.
- That a passing placebo test licenses the estimate. It falsifies the assumption when it fails and does not confirm it when it passes, because the assumption is about a period the diagnostic cannot see.
- That interrupted time series is never appropriate. It is reasonable for abrupt, well-dated changes where nothing else moved, which is a real situation and not this one.
- That any of this substitutes for an experiment. Where randomization is possible it removes the assumption instead of testing it, which is a different and much stronger position.
What to test next
Before choosing an estimator, write down how treatment was assigned and who decided. If the answer involves expected performance, the parallel trends assumption is already in trouble and the diagnostic will usually confirm it.
Then run the placebo before running the analysis. Split the pre-period, estimate the effect of the launch that had not happened yet, and only proceed if that number is close to zero. It is the cheapest step in the whole process and the one most often skipped, because it can only ever deliver bad news.
Where the rollout has already happened in waves, the estimator changes as well as the diagnostic. The two-way fixed effects regression above uses early adopters as controls for late ones and misreads any effect that grows after adoption; when the rollout was staggered shows the size of that error and what to run instead.
Where a rollout is still being planned, the highest-value intervention is not statistical. Ask for the order to be randomized among the markets that were going to be treated anyway. It costs nothing, it converts every design above into the right-hand column of the table, and the geo holdout example is what that looks like carried through.
How the two worlds were built Code
60 markets over 52 weeks of log conversions. Each market has its own level and its own weekly growth rate, and all of them share a seasonal shock. That is what makes a before-and-after comparison hopeless and a control group necessary. A true lift of +6.0% is applied to 15 markets for the final 12 weeks.
The two worlds share everything except assignment. In one, the 15 markets with the highest underlying growth rate are treated. In the other, 15 are drawn at random. Synthetic control weights are fitted by exponentiated gradient on the simplex, so weights stay non-negative and sum to one and the synthetic unit cannot extrapolate beyond its donors.
Every figure is the mean of 50 replications of the whole world rather than a bootstrap inside one. Bias is a property of repeated draws, and an interval around a single biased estimate looks tight and reassuring while being centered on the wrong number. These are not measurements from a real company.