When the Rollout Was Staggered
Markets adopt in waves and the effect grows after adoption. The regression everyone runs on that panel uses early adopters as controls for late ones and understates the effect by a third. What to run instead, and the plot that shows whether it can be believed.
The observational designs lesson opens on a rollout that went region by region and then estimates it as if every treated market had switched on the same day. Real rollouts do not work that way. Stores get the new program in waves as operations can staff them. Partners launch when their calendars allow. A pricing change reaches workspaces as their contracts renew. By the time anyone asks what it did, some units have been treated for six months, some for six weeks, and some not at all.
The regression almost everyone runs on that panel is the same one as before: market fixed effects, week fixed effects, and one dummy that is one when a market is treated. It has a specific failure on staggered data, the failure is large when the effect grows after adoption, and it was not widely understood until a run of papers between 2018 and 2021 worked it out. This lesson shows the failure on simulated data, shows the estimator that avoids it, and shows the diagnostic that tells you whether either can be believed.
The panel
64 markets over 44 weeks. Four cohorts of 12 markets adopt at weeks 12, 18, 24, 30, and 16 never adopt. Adoption order is random here, which sets aside the selection problem the previous lesson was about so that this one can be seen on its own.
The effect is simulated two ways. In the growing world it ramps with time since adoption, from +1.0% in the first week to +8.3% after twelve weeks and +11.1% after twenty-four, the shape of a program whose audience builds or whose algorithm learns. In the flat world it is +6.2% from the first week onward.
The target in both is the same: the average effect across every treated market-week the panel observed. In the growing world that average is +7.3%, lower than the long-run effect because many of the observed weeks are early ones.
Two estimators
Static two-way fixed effects is the regression above. With one adoption date it is exactly the difference-in-differences everyone intends. With several, it is a weighted average of every two-by-two comparison the data allows, including comparisons in which a cohort that adopted earlier serves as the control group for a cohort that adopted later. If the early cohort’s effect is still growing during the late cohort’s window, that growth is subtracted from the late cohort’s estimate as if it were a common trend. Some of the implied weights can be negative. A growing effect can be estimated as a shrinking one.
Cohort by time refuses those comparisons. For each cohort and each week, it compares that cohort’s change since the week before its adoption with the same change among markets that were not treated in either period, either never-treated markets or ones that had not adopted yet. Each of those is a clean two-by-two. The overall estimate is then an explicit average of them, weighted by cohort size, and the event-study path is the same cells averaged by weeks since adoption instead. This is the estimator of Callaway and Sant’Anna; Sun and Abraham’s interaction-weighted regression, stacked regression by cohort, and the imputation estimator of Borusyak, Jaravel, and Spiess all reach the same place by different routes.
Show the data
| Estimated lift (%) | Point | 95% CI |
|---|---|---|
| Static two-way fixed effects | +4.8% | +4.1% to +5.5% |
| Cohort by time, never-treated controls | +7.9% | +5.1% to +10.8% |
| Cohort by time, not-yet-treated controls | +7.8% | +5.2% to +10.3% |
Show the data
| Estimated lift (%) | Point | 95% CI |
|---|---|---|
| Static two-way fixed effects | +6.0% | +5.2% to +6.8% |
| Cohort by time, never-treated controls | +5.6% | +3.4% to +8.0% |
| Cohort by time, not-yet-treated controls | +6.0% | +3.7% to +8.2% |
Why it goes wrong, in one sentence
The static regression assumes one number describes the effect for every cohort in every week since adoption. When that is true, every two-by-two in the data estimates the same quantity and averaging them any way you like is fine. When the effect depends on time since adoption, the two-by-twos that use an already-treated cohort as the control estimate the late cohort’s effect minus the early cohort’s growth, and the regression averages those in with the rest without telling you.
The size of the damage depends on how much of the effect’s growth falls inside the window where an early cohort is serving as someone’s control. Here the effect roughly doubles between week six and week twenty-four after adoption, the early cohorts are still ramping when the later ones switch on, and the regression comes in 34 percent low with an interval that does not reach the truth. That is a confident wrong answer, which is worse than a wide right one.
The plot to read first
The event-study path is where a staggered design earns or loses its credibility. It shows the cohort-by-time estimate at each week relative to adoption, including the weeks before it. Those pre-period coefficients are placebo estimates: the effect of an adoption that had not happened yet, against the same controls. If they sit near zero, the controls were moving with the cohorts before treatment and the parallel-trends assumption has survived the only test available to it. If they trend, the post-period coefficients inherit that trend and nothing downstream can be trusted.
- Cohort-by-time estimate
- True effect
Show the data
| Weeks since adoption | Cohort-by-time estimate | True effect |
|---|---|---|
| -6 | 0.6 | 0.0 |
| -5 | -0.4 | 0.0 |
| -4 | 0.5 | 0.0 |
| -3 | 0.3 | 0.0 |
| -2 | -1.0 | 0.0 |
| -1 | 0.0 | 0.0 |
| 0 | 0.5 | 1.0 |
| 1 | 0.5 | 1.9 |
| 2 | 3.5 | 2.7 |
| 3 | 3.9 | 3.5 |
| 4 | 3.9 | 4.2 |
| 5 | 5.6 | 4.8 |
| 6 | 5.8 | 5.4 |
| 7 | 7.5 | 6.0 |
| 8 | 7.0 | 6.5 |
| 9 | 7.2 | 7.0 |
| 10 | 6.6 | 7.5 |
| 11 | 6.5 | 7.9 |
| 12 | 9.1 | 8.3 |
| 13 | 8.2 | 8.6 |
The path is also the answer to a question the average cannot answer. A program whose effect is +1.0% in its first week and +11.1% six months later is a program that should be judged after six months and not after six weeks, and a readout that reports the average over whatever mix of cohort ages the panel happened to contain is reporting a number that depends on when you asked.
What to run
Most statistical software now has this built in, and the choice among the implementations matters less than using one of them.
- In R,
did(Callaway and Sant’Anna) is the reference implementation of the cohort-by-time estimator and produces the event-study plot directly.fixest::sunabgives the Sun and Abraham version inside a regression you already know how to run, anddidimputationgives the imputation estimator. - In Stata,
csdidandeventstudyinteractcover the same ground. - In Python,
differencesimplements Callaway and Sant’Anna. Where it is not available, stacking is the estimator you can build by hand: for each cohort, cut a dataset of that cohort and its not-yet-treated controls over a fixed event window, stack the datasets with a cohort identifier, and run the fixed-effects regression with cohort-by-week effects. bacondecompin R or Stata shows you the weights the static regression actually applied, including the negative ones, which is the fastest way to convince a stakeholder that the number they have been quoting is a strange average.
Whichever you run, report the event-study path before the average, state which units served as controls, and say what the average is an average over. An average effect over “treated market-weeks in the panel” is a different quantity from “effect at twelve weeks” and from “long-run effect”, and the staggered design can produce all three if it is asked clearly.
How the panel and the estimators were built Code
64 markets with log-normal baselines and a shared seasonal curve
over 44 weeks, independent log-normal noise per market-week, and
Poisson counts. Four cohorts of 12 adopt at weeks
12, 18, 24, 30; 16 never adopt. The growing effect is
0.12 · (1 − exp(−(e + 1) / 12)) in log points at e
weeks since adoption; the flat effect is 0.06 log points throughout.
Static two-way fixed effects is the coefficient on the treated indicator after removing market and week means, which is exact on a balanced panel. Cohort by time computes, for each cohort and week, the cohort’s mean change from the week before adoption minus the same change among controls, then averages post-adoption cells weighted by cohort size; the event-study aggregates the same cells by weeks since adoption, shown only for the event times every cohort reaches. Intervals are percentile intervals from 300 market-level bootstrap draws stratified by cohort. All effects are reported as percentages by exponentiating the log estimates.
Context
Best suited for
Any effect that arrived in waves: store programs, partner launches, feature flags rolled out by account, pricing changes applied at renewal, and channel launches that went region by region.
Data requirements
- The adoption date for every unit, from the system of record rather than from memory, since a misdated cohort contaminates every comparison it enters
- Units that had not adopted during the window, or never did, to serve as controls; a rollout that reached everyone at once by the end of the panel has no controls for its final weeks
- A pre-period long enough for the placebo coefficients to mean something, which is at least as long as the post-period you intend to read
- The reason for the adoption order, written down, because it decides whether the parallel-trends assumption is plausible at all
What changes by context
- Local / omnichannel
- The canonical case. Store rollouts are staggered by operations, effects grow as staff learn the program, and the later waves are often the stores that needed it most, so both the weighting problem and the selection problem are usually present.
- Partner-led
- Partners launch on their own calendars, which staggers adoption for you. The controls are partners who have not launched, and the question to ask is why they have not, since that answer is the identification argument.
- Product-led
- Feature flags rolled out by workspace over weeks are a staggered design whether or not anyone called it one, and adoption effects on usage almost always grow. The fix is usually to randomize the rollout order, which converts the whole problem into an experiment.
When it breaks down
- Rollouts with no untreated units left by the end, where the last cohort has no controls for its later weeks and the estimator can only report what it can identify.
- Treatments with spillovers across units, such as a regional launch that lifts branded search in neighboring regions, where the not-yet-treated are partly treated.
- Very few cohorts or very small ones, where the cohort-by-time intervals are too wide to be useful and the static regression’s precision is tempting for the wrong reason.
What this does not establish
- That two-way fixed effects is always wrong on staggered data. In the flat world it was right and more precise than the alternative. It is wrong when the effect varies with time since adoption, which is the common case and the one that cannot be assumed away.
- That the cohort-by-time estimator identifies the effect. It aggregates valid comparisons correctly; whether the comparisons are valid depends on how adoption was ordered, which this simulation made random and real rollouts do not.
- That the size of the bias transfers. A third is a property of how fast this effect ramped relative to the spacing of the cohorts. The direction is general; the size is not.
- That the event-study path is a clean picture of dynamics. It is shown only for event times every cohort reaches. Beyond that, the path is a changing mix of cohorts, and a turn in it can be composition rather than effect.