The Long-Run Holdback

The quarterly review listed 14 shipped wins on the growth surfaces with lifts that summed to 38% on activation. Activation for the quarter was up 9%. Everyone agreed the difference was seasonality, or the market, or attribution between tests, and nobody could say which, because every one of the 14 tests had closed its control arm the day it shipped.

There was no group of visitors left who had seen none of the changes, so there was nothing to compare the quarter to.

The decision it encodes

What did the quarter’s shipped changes add up to, and which proxies earned their place?

Every test in this system measures one change against its own control. This component measures the program against a control, which is a different thing and the only thing that answers the question the review was asking.

A control for the program, not the test

A holdback is a small share of visitors, drawn once and held, who are excluded from every test and never receive any change the growth program ships during the holdback’s period. They see the growth surface as it was at the start of the period. Everyone else sees the surface as the program has made it. The difference between the two, on the outcome the program is for, is what the program did.

“Never receive any change” has a contract behind it. Security patches, legal requirements, accessibility fixes, and infrastructure migrations reach the holdback too, and the contract lists them, because withholding them is not a measurement decision anyone should make. What the holdback is withheld from is the set of changes the program chose to ship, and every exception is recorded with what it might have done to the comparison.

It is the same construction as the lifecycle system’s persistent holdout, for the same reason: a program-level effect that survives every individual change, read on a schedule, without a negotiation. The difference is what it is read against. Lifecycle reads its holdout on the campaign outcomes. This one is read on retention and revenue, the rungs no single test could reach, and it is read on the proxies too, which is how the proxies are scored.

What it costs, and how to size it

The holdback forgoes every win for the visitors in it, so its cost is its share of the program’s incremental benefit, not its share of the business. At 5% of visitors and a program that adds a tenth to activation over a year, the cost is half a percent of activation, standing. That is the price, it is real, and it is worth stating in those terms to whoever approves it.

The size is a power question at the program level, and 5% is an illustration rather than a rule: the right share is the smallest one that can detect the smallest cumulative effect the program plausibly has. A 5% holdback on a surface with a hundred thousand visitors a month is five thousand visitors against ninety-five thousand, which reads a program effect of a few percent on activation over a quarter and reads retention over two. Smaller programs need a larger share or a longer read, and a holdback too small to detect the program’s plausible effect is a cost with no return, which the geo holdout designer will show for market designs and the same arithmetic shows here. It is also why the holdback is a later stage of this system rather than an early one: it needs enough shipped changes to reconcile, enough traffic to read, and the engineering to keep an old experience alive, and a program that has none of those yet has its tests’ own controls as its evidence.

What the quarterly holdback readout reports
Line	What it decides
Program effect on the outcome	Retention and value per visitor, holdback against everyone else, with an interval, read on fixed-maturity cohorts: D90 value among entrants assigned at least ninety days ago, not a mix of day-two and day-two-hundred members. The number the review was asking for.
Program effect on each proxy	Activation and paid conversion the same way, so the proxies can be checked against the outcome in aggregate rather than test by test.
Bottom-up forecast from the registry	Each shipped effect translated onto the same scale as the holdback read, incremental value per eligible entrant under the reach it actually had, then summed, shrunk. Relative lifts on different surfaces, populations, and horizons cannot be added as they stand.
Reconciliation residual	The bottom-up forecast beside the top-down holdback read on the same metric. The gap is the finding; what caused it is the next question.
Holdback integrity	How many holdback visitors were exposed to a shipped change anyway, which is the leak, and how many left the holdback through identity changes.
Proxy verdicts	Which proxies’ aggregate lifts tracked the outcome’s and which did not, feeding the metric ladder.

The residual is the line that changes behavior. A program whose bottom-up forecast is four times what the holdback measures has learned something, and it learns it here or nowhere. What it has not learned is why: the winner’s curse, interactions, decay, reach that fell short of the plan, production differing from the test, leakage, and a proxy that stopped predicting are all candidates, and the holdback names the gap without naming the cause.

Why it has to rotate, and why slowly

A holdback held forever becomes a museum. Its visitors see a surface that drifts further from the product every quarter, they cannot be given features that depend on shipped changes, and after a year the comparison is between the product and an archive of it. Rotation fixes that: at a set interval the holdback is released, receives everything that shipped, and a fresh holdback is drawn.

The interval is a trade. Rotate quarterly and the holdback can only ever read a quarter’s worth of changes, which is fine for the sum-of-wins check and too short for retention. Rotate yearly and retention is readable and the museum problem is real by month nine. Most programs land on a rolling design: two holdbacks offset by half a period, so one is always old enough to read retention while the other is fresh enough to be a fair control.

Maturity, honestly staged

Stage 1: a holdback exists. Drawn once, excluded from every test, and read on activation at the end of the quarter. The program has a number for itself.

Stage 2: the bottom-up forecast beside it. The registry’s shrunk effects are translated onto the holdback’s scale, summed, and the residual is reported. The shipping rule gets its first grade from outside itself.

Stage 3: read on retention and revenue, with rotation. The rolling design lets the outcome be read without the control turning into an archive.

Stage 4: proxies scored from the holdback. Each proxy’s aggregate lift is compared to the outcome’s, and the metric ladder changes when the evidence does. The loop is closed.

Sizing, the rolling design, and reading proxies against the outcome Technical

The holdback readout is a two-arm comparison with arms of very different size, so its standard error is dominated by the small arm: roughly the outcome’s standard deviation over the square root of the holdback count. Doubling the holdback improves precision by about 40% and doubles the cost, and the useful size is the smallest one whose detectable program effect is below the effect the program plausibly has. A program that adds two percent a year needs a holdback that can see two percent, or it should not pay for one.

The rolling design draws holdback A at the start of the year and holdback B six months later, releases A at month twelve and draws a new A, and so on. At any readout one holdback has at least six months of exposure to the program’s changes, enough to read retention, and one has at most six, close enough to the current product to be a fair control for the sum-of-wins check. Both are read; they answer different questions.

Proxy validation from the holdback compares the program’s effect on each proxy to its effect on the outcome. A proxy that moved by a tenth while the outcome moved by a fiftieth has been optimized past what it predicts. This is the aggregate counterpart of the per-test validation in the metric ladder, and the two disagree exactly when the tests have been selecting changes that move the proxy for its own sake, which is the failure this component exists to catch.

Context

Best suited for

Programs that ship more than a handful of changes a quarter on the same surfaces, report their wins as a sum, and cannot currently say what the program did to retention or revenue.

Data requirements

A holdback membership held in the same store as test assignments, excluded from every flag by rule
Exposure logging on every shipped change, so leaks into the holdback are measured
Retention and revenue per visitor at the horizons the program cares about, joinable to holdback membership
The registry’s shrunk effects per period, to sum against the program effect

What changes by context

Product-led: The natural home and the case with the most to reconcile, since the program ships constantly and the sum of wins is usually a large number nobody quite believes. The holdback is what makes that number arguable.
B2C subscription: Read at the renewal. The holdback is the only place a pricing program can see what its year of tests did to twelve-month retention, and the rolling design is what makes a twelve-month read possible without a twelve-month museum.
B2B SaaS: At the account level the holdback is a few dozen accounts and it can read almost nothing on its own. It still earns its place as the leak check and the sum-of-wins reconciliation, read over a year rather than a quarter.

When it breaks down

Programs that ship a change or two a quarter, where the holdback costs more than the reconciliation is worth and the tests’ own controls suffice.
Surfaces where visitors cannot be held on an old version, because a shipped change is a legal, security, or platform requirement that must reach everyone.
Traffic too low for a holdback of any tolerable size to detect the program’s plausible effect, where the program’s total is unmeasured and should be reported as such.