Template Split Design

The title pattern changed across every product page on the 14th. Over the following month, organic sessions to those pages rose 6%.

So did the category. A competitor went down for a week. And a core update landed on the 20th. The 6% is real, it is a fact about what happened, and there is no arrangement of that data that separates the change from the other three things, because everything moved at once and nothing was left alone.

The decision it encodes

Which pages get the change, and which stay behind as the comparison?

Organic has one structural disadvantage and one structural advantage. It cannot be held out at the visitor or channel level, so there is no holdout in the ordinary sense. It has many near-identical pages, so the comparison can live inside a single template.

The control is the other half of the template

This is the design, and it is the whole reason organic is a separate system. A lifecycle program withholds a person. Performance withholds a market. Organic withholds the change: half the pages of a template ship it, half do not, and the difference between them is the estimate.

That constrains what can be tested. Anything sitewide, a navigation change, a migration, a speed improvement in shared code, has no other half and needs the modeled counterfactual instead. What splits cleanly is anything expressed per page whose effect stays on that page: titles, headings, schema, content blocks, image handling.

Internal links sit between the two. A global navigation change is sitewide. A related-links module on a template can be tested, but the unit is no longer the page: the pages that gain links change too, so source and destination pages are assigned together as a cluster and the outcome is read across the cluster. The same is true of any pages that compete for the same queries, which is why the unit of randomization is sometimes a topic, a category, or a city rather than a URL.

This is not landing page testing

Worth separating, because the phrase invites the wrong picture. A conversion test splits visitors arriving at one page, controls its own assignment, exposes people the moment they land, and measures what they do next.

This splits pages, and the visitor may never arrive at all. The outcome is whether the search engine sends anybody, not what happens once it has. You cannot serve one version to the crawler and another to a person, so the change is real for everyone who sees it. And assignment is to URLs rather than to people, which is why a template with four thousand pages is a far smaller experiment than a landing page with four thousand visitors a day.

What running one involves

Three pieces, and the first is what rules the design out for most teams.

Assignment is on the URL, and every visitor to that URL sees the same thing. Hash the canonical URL with a salt, bucket it, and serve accordingly. This is not politeness: varying the response by user agent so that the crawler sees one version and a person sees another is cloaking, and it converts a measurement project into a ranking penalty. It is also the precise reason a conversion testing tool cannot run this. Those assign per visitor, which is the wrong unit and the wrong mechanism.

The variant is applied where the HTML is assembled, which in practice means one of two places. At the edge, in a Cloudflare Worker, a Lambda@Edge function, or Fastly, rewriting the response for URLs in the treated bucket. That is how the commercial tooling in this space works, SearchPilot being the established one, and how most large sites run it. Or in the application, if the CMS can carry a per-page flag and templates can branch on it, which is easier to reason about and usually harder to get engineering time for.

The outcome comes from Search Console, per page, per day. Clicks and impressions by URL through the Search Console API, joined to the arm. Analytics sessions work and are noisier, since they mix in traffic that did not come from search. Either source is then aggregated per arm into a daily series and read as an interrupted time series, for the reason in the appendix below: per-page traffic is far too volatile to difference directly. Google’s CausalImpact, or any Bayesian structural time series in the same family, is the usual estimator.

Stratify, because the pages are nothing like each other

Random assignment across a template with a long tail will, often enough to matter, put a disproportionate share of the traffic in one arm. With forty pages carrying half the sessions, one unlucky draw dominates the result.

The fix is ordinary: sort pages by traffic, form strata, randomize within each. That balances the large pages by construction rather than by luck, and it is worth more here than in almost any other design on this site, because the concentration is more extreme than anywhere else.

Query competition is the visible half of that and the manageable one: it is worst where pages chase identical terms, which the inventory can flag before launch, and mildest on long-tail templates where each page owns its own phrases.

The half you cannot design around is that search engines evaluate at the level of the site and the template, not only the page. A change applied to half a template can move how the whole template is assessed, which means the control arm is not untreated in the way a holdout customer is untreated. Nothing in the design fixes that. What it does is bound it: a positive result on a split is weak evidence that the same change applied everywhere would return the same amount, and it is strong evidence that the change is not harmful, which is usually the decision being made.

Set the window before launch

A lifecycle test reads in days. This one starts when the change is re-indexed, which happens page by page over weeks, and the ranking effect keeps moving after that with no observable moment at which it stops. So the readout window is a decision made before launch, from the template’s historical volatility, the effect worth detecting, weekly seasonality, and how much of the treated arm the crawl is expected to reach. Several weeks is common and nothing is universal: the commercial tooling reads many tests at two to four weeks, and a slow-crawled tail can take months. What the window is not is a date chosen after looking at the series. A window that begins at deploy is measuring a mixture of treated and untreated pages, and the mixture is not random. Crawl and index state is the component that makes this tractable.

Maturity, honestly staged

Stage 1: any split at all, randomized. Most organic changes ship to everything, so a template with a real control arm is already a large step.

Stage 2: stratified by traffic. The large pages stop deciding the result by chance, which on a concentrated template is the difference between a design and a coin toss.

Stage 3: pre-period balance checked before launch. Divergence gets caught while it is still cheap to redraw the assignment.

Stage 4: assignment recorded and stable across deploys. A page stays in its arm through a release, a re-crawl, and a CMS migration, so the test survives ordinary engineering.

Power on a concentrated template Technical

In practice the readout is not a difference in sessions per page. Page-level traffic is far too volatile for that: a single page can lose most of its sessions to a change in what the results page looks like, with nothing happening to the page at all. What works is to sum each arm into a daily series, fit the relationship between the two over the pre-period, and read the effect as the divergence afterwards, which is the same interrupted time series the counterfactual component uses with a far better control.

Its precision is bounded by the effective number of pages rather than the page count, for the reason set out in the inventory: with session shares wᵢ, the effective count is (Σwᵢ)² / Σwᵢ², and on a concentrated template it can be an order of magnitude below the number of URLs. That is the screen. The detectable effect itself comes from placebo splits: divide the template’s history into two random halves, stratified the same way, run the planned estimator with a fake launch date, and repeat until the spread of the estimates it returns when nothing happened is known. That spread is the test’s noise floor, and it already contains the autocorrelation, seasonality, and interference that no formula does.

Two things raise it materially. Stratifying by traffic decile removes the between-strata variance from the comparison, which is where most of the variation lives. And using each page’s own pre-period as a covariate, the same variance reduction the lifecycle readout uses, helps more here than almost anywhere, because a page’s traffic is strongly autocorrelated and the pre-period is therefore highly predictive.

Excluding the head pages is a legitimate design and changes the estimand to the effect on the tail. That is frequently the right choice, and it has to be reported as what it is rather than as the effect of the change.

Context

Best suited for

Templates with enough near-identical pages that half of them can be left alone without anybody noticing, and where the change is expressed per page rather than sitewide.

Data requirements

A template map and per-page traffic, to form strata and to compute what the split can detect
A way to serve a variant by URL: edge compute such as Cloudflare Workers or Lambda@Edge, or a per-page flag the CMS and templates both respect
Per-page clicks and impressions at daily grain, from the Search Console API rather than from analytics where possible
A stable page identifier that survives a re-crawl and a CMS release

What changes by context

B2C ecommerce: Product pages are the natural surface and they churn: pages go out of stock, get delisted, or are recreated under new URLs mid-test. Attrition is not random, so the population has to be fixed at assignment and dropouts reported rather than removed.
Marketplace: The best case on the site. Listing templates run to millions of pages with a genuine long tail, so the effective count is high and cannibalization between arms is low because each listing owns its own queries.
Local / omnichannel: Location pages split well and compete with each other geographically rather than by query, so interference between arms is unusually mild. Watch for pages in the same city, which do compete.

When it breaks down

Sitewide changes, which have no other half and need a modeled counterfactual instead.
Templates of a few high-value pages, where the effective count is too small to detect anything a business would act on.
Pages that compete for identical queries, where an improvement to one arm comes out of the other and the difference overstates what a full rollout would do, unless the competing pages are assigned together as a cluster.
Sites where engineering cannot target a subset of pages, in which case the design is unavailable regardless of its merits.