Counterfactual Traffic

The migration went live in March. Organic sessions fell 11% and recovered over seven weeks.

Everyone has a view. The engineers say the recovery proves the migration was fine. The agency says the dip proves it was not. Somebody points out that March is seasonally weak and that a core update landed in the middle of the recovery. All three are describing the same line on the same chart, and none of them has a comparison.

The decision it encodes

What would these pages have done if we had changed nothing?

When a split is available, the other arm answers this by existing. For most sites and most changes it is not available, so this is the ordinary case rather than the fallback: sitewide work has no other half, and a site without a template of enough comparable pages has no split to run at all.

What cannot be split, and therefore lands here

Two categories, and the second is larger than the first.

Changes with no other half. Global navigation and site-wide link architecture, because a change to the whole link graph reaches pages that were not themselves changed. Site migrations. Rendering and speed work in shared code. A related-links module on one template is the exception: it can be split when source and destination pages are assigned together as clusters, which the split design describes.

Sites with no other half. A template needs enough comparable pages and traffic for two halves to track each other before a split can detect anything, and most sites do not have one. For them every change lands here, including the ones that would split beautifully somewhere larger.

Build it from a series the change did not touch

A forecast from a page’s own history is the weakest version, because it carries no information about what happened in the world during the test window, which is the period in question.

The stronger construction uses a comparison series: an untouched template on the same site, the same pages’ performance on a search engine you did not change for, or a demand index such as query volume for the same terms. The estimate is then the divergence between the treated series and its comparison, which is an interrupted time series with a control rather than a before-and-after.

A core update inside the window is a test of the comparison

Search engines change ranking systems several times a year. The larger updates are posted on the engine’s status dashboard as they begin rolling out, not before, and they are not evenly distributed across sites or separable from anything else happening at the same time.

For a split, an update is one of the reasons the control arm exists: two halves of one template drawn at random are hit by the same update, and the comparison absorbs it. For a modeled counterfactual it is more dangerous, because the treated series and the comparison series can be hit by different amounts, and the fitted relationship between them may not survive.

So an update inside the window is flagged, and then tested rather than assumed fatal. If the gap between the treated series and its comparison opens at the update date rather than at the change date, or a placebo run on the update date returns an effect, the identifying relationship broke and the estimate is reported as inconclusive for that window. What is not acceptable is either of the two shortcuts: controlling for the update with a variable nobody has, or quietly extending the window until the number looks stable. The second is how these analyses lose their credibility.

Say the horizon, and mean it

Organic effects accrue over months. A readout at four weeks measures crawl coverage as much as ranking, and a readout at three months measures the change plus everything else that happened in three months.

Neither is wrong and they answer different questions, so the horizon belongs in the estimand rather than in a footnote. The practical form is to report the same estimate at several horizons, which distinguishes an effect that is still building from one that has already reversed.

Maturity, honestly staged

Stage 1: a before-and-after with the seasonal pattern stated. Weak, and better than a chart with a vertical line on it, because it is explicit about what it assumes.

Stage 2: a comparison series, chosen and justified. The estimate becomes a divergence rather than a change, and it survives a category-wide movement.

Stage 3: the relationship fitted on a pre-period and validated out of sample. The counterfactual is testable rather than asserted.

Stage 4: a core update register, and windows tested when one lands. The program can say a result is inconclusive because the comparison broke, and show the check that says so, which is what stops the method being used to prove whatever was wanted.

Interrupted time series, and what a synthetic control adds Technical

The basic form regresses the treated series on the comparison series and a seasonal term over the pre-period, projects that relationship through the test window, and reads the effect as the gap between actual and projected. The interval has to account for the projection uncertainty as well as the residual noise, which is the step most implementations skip and the reason these estimates are usually reported far too confidently.

A synthetic control generalizes the comparison: instead of choosing one series, fit a weighted combination of several untouched ones that best reproduces the treated series over the pre-period, and use that combination as the counterfactual. It is stronger where no single comparison tracks well and it needs a long pre-period, which organic data usually has.

Both rest on the same assumption, that the fitted relationship would have continued, and neither can test it for the period that matters. The diagnostics that earn belief are pre-period fit, out-of-sample validation on a held-back window, and placebo runs on dates where nothing happened. A method that returns an effect on a placebo date is telling you the design cannot support the one you intend to run.

Context

Best suited for

Changes that reach every page and therefore have no on-site control: migrations, navigation, shared rendering, and anything that alters the internal link graph.

Data requirements

• Daily sessions or clicks by template, for a pre-period long enough to fit seasonality against
• At least one comparison series that the change plausibly did not touch
• A dated register of deploys, migrations, and known ranking updates
• Query volume or another demand measure, to separate a ranking movement from a market one

What changes by context

B2C ecommerce

Promotions and seasonality dominate the series, and both move demand and rankings at once. Query volume for the same terms is the most useful comparison here, because it absorbs the demand movement the site cannot.

Marketplace

Supply changes the page inventory continuously, so the treated series moves for reasons that have nothing to do with the change. Holding the page set fixed at the start of the window matters more than the choice of comparison.

B2B SaaS

Volumes are low enough that weekly noise swamps most effects, so the horizon has to be long and the verdict is often that the change cannot be measured at all.

When it breaks down

• Windows containing a core update that broke the fitted relationship, where the shock hit the two series by different amounts and nothing separates it from the change.
• Sites with no untouched comparison, where the estimate reduces to a forecast from the past and should be labeled as one.
• Changes shipped alongside other changes, which is the normal case and the reason a deploy register is a prerequisite rather than a nicety.
• Short pre-periods, where seasonality cannot be estimated and the projection is a straight line with a confident interval around it.