Organic Search

The quarterly review has a slide that says organic sessions grew 11%, and a team that shipped forty content changes in the same quarter. Nobody can say which of the forty did it, whether any of them did, or whether the 11% would have arrived had the team spent the quarter on holiday.

Nothing on the slide is evidence. The category may have grown, a competitor may have slipped, the season may have turned, and the search engine ships ranking changes several times a year, announced on its status dashboard as they begin and not before. The 11% is a fact about what happened. It is not a fact about what the team did.

Every other channel can be switched off. This one cannot

That is the whole reason this is a separate system rather than a chapter of another one.

A lifecycle program can withhold a message from a random slice of its list. Paid media can go dark in a set of markets. Neither is comfortable, and both are possible. You cannot stop ranking. There is no version of the next six weeks in which your pages are absent from search results and you get to see what happens.

So organic cannot be held out at the level every other system uses, the person or the market. What it can withhold is the change itself: a random set of eligible pages, or clusters of pages that link to and compete with one another, that do not receive it. Everything here exists to build that comparison, or to model one where nothing could be withheld. Those are the two families, a controlled rollout and a modeled counterfactual, and most of the machinery is about knowing which one you are in.

It is also the slowest loop in the section, and it is better to say so than to draw the dashed arrow and hope. A split reads out when the treated pages have been re-crawled and a window set before launch has passed, which is weeks the analyst does not control, and only a template with enough comparable pages and traffic for two halves to track each other can be split at all. On most sites the loop closes a few times a year. The machinery earns its place by making those few readouts trustworthy, and by keeping a roadmap and a change log so that the modeled counterfactual, which is what most changes will get, has something to stand on.

The four stages, in plain terms

Content inventory is the part nobody wants and everybody needs. You cannot measure a change to “the product pages” until somebody can say which pages those are, which template renders them, and which of them are in the index at all. It also records how concentrated the traffic is, because a template where forty pages carry half the sessions cannot support the design in the next stage, however many URLs it has.

Design is where the comparison comes from. If a template has enough comparable pages, ship the change to a random half of them and leave the other half alone: the untouched half is the control, and seasonality and ranking updates hit both arms together. Alongside it sits the awkward fact that shipping is not delivering. A search engine re-crawls pages over weeks, in an order that is not random and tends to favor popular, fresh, and well-linked pages, so the change lands on the big pages first and on the tail possibly never.

Estimation reads whichever comparison you have. Where a split exists, the other arm answers the question. Where it does not, and for most sites and most changes it does not, the counterfactual has to be modeled from a series the change did not touch. Then one accounting step that gets skipped almost everywhere: some of an organic gain on terms you also buy may be your own paid traffic arriving free, and the number the business needs is the change in total search, not two channel reports that can add to more than what arrived.

Roadmap decides what changes next. Audits produce hundreds of items ordered by severity, which grades conformance rather than consequence. Ordering by traffic at stake instead promotes dull changes to large templates and demotes dramatic findings on pages nobody visits.

The loop is what makes it compound

The dashed arrow is the only edge that is not a build dependency. A change shipped to a whole template produces an outcome nobody can attribute. The same change shipped to half a template produces the same outcome and an estimate, and that estimate is what prices the next change of the same kind.

Nobody knows what a schema change is worth on a first attempt. After four splits on four templates, the fifth is ranked on what this site returned rather than on what somebody claimed at a conference. The value is not any single test. It is that the ordering gets less wrong every quarter.

Where to start when none of it exists

First, the inventory. Not because it is exciting, but because every other component takes it as input and none of them can be faked without it. A URL to template mapping and traffic per page is a week of work and it is the difference between a roadmap and a list.

Then a comparison for one change. Either a split on the largest template that can support one, or a modeled counterfactual with a comparison series chosen and justified. One change measured properly is worth more than a quarter of shipped work nobody can attribute.

Then crawl logging. It costs nothing once the log pipeline exists and it changes how every subsequent readout is read, because the share of pages re-crawled turns a disappointing result into a diagnosis.

Then the paid subtraction, and the roadmap last. The roadmap is only as good as the estimates feeding it, so it comes after there are some.

What this leaves out

On-page conversion testing. This system randomizes pages, so it can attach downstream conversions and value to organic landings by arm and report the total business effect of a change, traffic and visitor mix together. What it cannot do is isolate the effect of a page on the people who reach it, which randomizes visitors and is a growth experiment.

Content production. Deciding what to write about is a demand question rather than a measurement one, and the scope of this system is what happens to pages that already exist. The roadmap says how a program keeps that from biasing it toward pages that already have traffic.