Crawl and Index State

The split launched on the first of the month. The readout compares the four weeks after it against the four weeks before.

Except the treated pages were re-crawled across a twenty-six day window. The pages that were re-crawled on day two are the ones that get crawled most often, which are the ones with the most traffic and the most links. The pages still carrying the old title on day twenty-five are the thin ones at the bottom of the template. So the treatment period is different for every page, and its length is correlated with the outcome.

Nothing in the analysis knew that, because the only date it had was the deploy.

The decision it encodes

When did the change reach the index, and for which pages?

This is the organic version of a question the lifecycle system asks about email: a message sent is not a message delivered, and the gap between them is not random. Here the chain is shorter and the same shape. A page changed is not a page re-indexed.

Crawl timing is non-random exposure

Search engines decide how often to fetch a page from its popularity, how often it changes, how the site is regarded, and how much the server will bear. Those are not the same as traffic, but they move with it, so the order in which treatment lands is correlated with the very variable you are trying to measure an effect on.

That makes crawl timing an exposure mechanism, not a confounder of the design. Random assignment still protects the comparison of assigned arms. What it does not protect is any analysis that conditions on which pages the engine happened to fetch, which is where the selection comes in.

Where a page can be between deploy and effect

State	What it means for the readout
Assigned	In the arm. This is the population the randomization applies to, and the one the estimate is about.
Served	The intended version is live for that URL. Nothing has happened yet as far as search is concerned.
Fetched	A verified crawler has requested the new version, per the server logs. The clock starts here, per page, not at deploy.
Index status observed	URL Inspection reports a new last-crawl time or a changed index or canonical state. Days after the fetch, and the closest thing to a delivery receipt the engine gives.
Window matured	The outcome window declared before launch has elapsed. There is no observable moment at which a ranking has finished moving, so the window is a decision rather than a measurement.

Analyze in event time, or accept the dilution

Two options, and they answer different questions.

Calendar time from deploy is intention to treat. It includes every assigned page whether or not the engine ever looked at it, and it estimates the effect of shipping the change, which is the decision a team is making. It is diluted by however many pages were never re-crawled, and that dilution is the finding rather than a flaw.

Event time from re-crawl aligns every page on its own day zero and asks what happened once the change was live. It is the sharper measurement and it conditions on something the search engine chose, so it is no longer protected by the randomization. Report it as a diagnostic, not as the headline.

Log it, because none of it is recoverable later

The record needed is short and nobody keeps it: for each page and arm, the deploy date, the date the new version was first fetched, and the date it first appeared in results.

The deploy date comes from your own release pipeline, at page grain rather than template grain. The fetch date comes from server or CDN logs filtered to verified search engine crawlers, which on Cloudflare or Fastly is a log push to object storage and a daily job; Search Console’s crawl stats report gives the same picture in aggregate and cannot be joined to a page, so it diagnoses and does not measure. The index date comes from the URL Inspection API, which returns the last crawl time, the index verdict, and the canonical the engine chose, and does not return the title or snippet being shown. Whether the new version is what appears in results needs a scheduled check against the live result, which is a separate and rate-limited job.

None of these can be reconstructed after the fact. Six weeks later the only thing left is that the change is live everywhere, which is the state in which every page looks identically treated.

Maturity, honestly staged

Stage 1: the deploy date recorded per page. Sounds trivial and is often absent, because changes ship by template and nobody writes down which pages the template covered that day.

Stage 2: re-crawl dates from log files, joined to the arm. The staggered rollout becomes visible, and the share of pages reached can be reported.

Stage 3: readouts in both calendar and event time. The headline stays intention to treat and the event-time view sits beside it as a diagnostic.

Stage 4: crawl coverage as a launch gate. A template whose tail is rarely crawled is identified before a test is designed on it rather than after it returns nothing.

Why the diluted estimate is still the useful one Technical

With assignment randomized and some assigned pages never re-crawled, the calendar-time difference between arms is an intention-to-treat estimate. It is unbiased for the effect of shipping the change, and it is smaller than the effect on a page that was re-indexed, by roughly the share that was reached.

The effect among reached pages can be estimated the same way it is for email, by dividing the intention-to-treat estimate by the share of treated pages whose new version had reached the index by the readout. Control pages are re-crawled at the same rate and have nothing new to show, so the denominator is the treated arm’s indexation rate rather than a difference in crawl rates between arms, which would be close to zero. This is a treatment-on-the-treated calculation and it carries that method’s assumptions: the change does nothing to a page the engine never fetched, reaching the index is observed correctly, and treated pages do not move control pages. The first is close to true for an on-page edit and false for anything that alters internal linking; the last is the interference the split design already warns about. Report it as a diagnostic under the intention-to-treat number, not as a recovered truth.

The practical reading is that the diluted number is the one a roadmap should use. It is what shipping the change to this template will do, on the crawl behavior this site gets, which is the quantity the next decision turns on.

Context

Best suited for

Any site large enough that a template change takes more than a few days to propagate, which is most sites above a few thousand pages.

Data requirements

• Server or CDN logs with search engine user agents, retained long enough to cover a test window
• Deploy records at page grain rather than at template grain
• The arm assignment, joined to both, on a page identifier that survives a release
• Some observation of what is being served in results, to distinguish crawled from indexed

What changes by context

B2C ecommerce

Crawl budget is spent heavily on faceted URLs that nobody wanted indexed, so the tail of real product pages is reached slowly. The share of assigned pages re-crawled is often well under half, and reporting it changes how the result is read.

Marketplace

Listings change constantly for reasons unrelated to the test, so re-crawls are frequent and the confound is smaller. The difficulty moves to separating the tested change from the ordinary churn of the listing itself.

B2B SaaS

Few enough pages that everything is re-crawled quickly, which removes this problem almost entirely and replaces it with the harder one of having too few pages to split.

When it breaks down

• Sites without accessible server logs, where crawl dates can only be approximated from third-party reporting at daily aggregate.
• Changes that alter internal linking, where pages are affected without being re-crawled themselves and the exposure model does not hold.
• Very small sites, where everything is re-crawled within days and the staggering is immaterial.