Growth Experiments

The pricing page test won. Signups rose 12% with an interval clear of zero, the result was announced in the Monday meeting, and the variant shipped that afternoon. Six weeks later paid conversion from those signups was down, refunds were up, and nobody connected the two, because the test had been read on signups and closed.

Nothing about the randomization was wrong. Everything after it was.

The visitor is the default unit, and that is why it is fast

Every system on this site is built around the unit the business will let you withhold. Here the default is an eligible visitor: someone on the signup flow, in onboarding, on the pricing page, at the paywall. Where the treatment spills over, to colleagues who must see one price, to a household on one device, to the other side of a marketplace, the unit moves up to the smallest thing that contains the spillover, and the sample shrinks with it. There are thousands of visitors a day, the change is a build rather than a spend, and nobody has to be persuaded to leave a market dark. Assignment is a flag, and its marginal cost is close to zero; what remains is the engineering, the maintenance of two experiences, and the visitors who see the worse one.

That is why this loop closes in a week where performance closes in a quarter, and it is why the section treats it as the reference: it is the loop the other four are trying to approximate under scarcer units. It is also why the failure modes are different. When randomizing is free, nothing stops a team running forty tests a quarter and shipping whatever won, and the rigor has to live somewhere other than in the difficulty of getting a test approved.

The hard parts are before and after the randomization

Before: who the visitor is. A test on the signup flow randomizes people who have no account yet, by a cookie or a device, and the outcome that matters arrives after they have one. If the assignment does not survive the moment a visitor becomes a user, the readout compares two arms that have quietly traded members, and nothing in the result says so. In a product bought by teams, the visitor is not even the right unit: two people in one company seeing two prices is a support ticket, not an experiment.

After: what the result is allowed to mean. A lift on signups says nothing about activation, a lift on activation says nothing about retention, and a lift on paid conversion says nothing about the value of what was sold. A test that reads out on the fast metric and ships is a machine for optimizing the top of the funnel at the expense of the bottom. Whether a test may launch alongside another, whether it may stop when it looks good, what happens to the guardrails when the primary metric wins, and what the year’s forty wins added up to are all decisions, and a team that has not made them in advance makes them in the meeting, differently every time.

The four stages, in plain terms

Surfaces and units is deciding which parts of the product this system covers, at what unit a visitor is randomized, and how that assignment survives signup. It sounds like configuration and it is where most contaminated results start.

Metrics and design is one metric that decides, a short list that can veto, and a record of how each proxy earned its place: activation is a stand-in for retention only if, across the tests that have run, lifts on one have predicted lifts on the other. It is also which tests may launch now, on which visitors, alongside which others, at what allocation and for how long, which is the decision a program makes forty times a quarter without noticing.

Readouts is when a test may stop and what it reports. A test read daily and stopped when it crosses a line is wrong about a quarter of the time on the schedule most teams run; the fix is a horizon set in advance or an interval that stays valid however often it is looked at, plus the checks that run before anyone reads a lift.

Shipping and learning is the rule that turns a readout into a decision, applied the same way to every test, the registry that records what was asked and learned, and, once the program ships enough to be worth reconciling, the holdback: a small standing share of visitors who never see any shipped change, so that once a quarter the sum of the wins can be checked against what the program added.

The loop is what makes it compound

The dashed edge runs from the holdback back to the metrics. Every test in the quarter shipped on a proxy, and the holdback is the one place the proxies are scored against the outcome they stood in for. A proxy whose test lifts kept predicting retention keeps its place. One that did not is demoted, and the next quarter’s tests are read on something better. That is how a fast loop stays a rigorous one rather than a fast way to be wrong.

Without that edge the program still runs, and it looks healthy: every quarter reports its wins, the wins sum to a number, and the number never meets the growth chart. The holdback is what makes the two meet.

Where to start when none of it exists

First, assignment that survives signup. Not a metric, not a dashboard. If a visitor’s arm can change when they log in, nothing downstream can be trusted, and this is the one thing most flag tools do not do by default.

Then a registry entry before every launch. A structured brief: the hypothesis, the unit, the metric, the guardrail bounds, the horizon, and the smallest effect worth shipping, written before the flag turns on. It costs a page and it is what every later component reads from. The registry’s clever parts, shrinkage and priors, come years later; the brief comes first.

Then one decision metric and two guardrails, with contracts. Activation as the metric, refunds per assigned visitor and one operational metric as the guardrails, each defined down to its denominator and window, agreed before the next test launches. Most of the discipline in this system is the discipline of deciding in advance.

Then runtime, layers, and a stopping rule. A runtime computed from the worthwhile effect, a layer scheme so two tests on one button never meet, and either a fixed horizon nobody peeks before or an always-valid interval. Either stopping rule is fine. Reading daily and stopping on significance is not, and it is what every team does until told otherwise.

Then the shipping rule. Five branches, decided once, applied to every readout without a meeting.

Then the holdback, once the program is large enough to reconcile. A standing slice costs its share of every win and buys the only number that says what the program is worth, and it is worth paying for once there are enough shipped changes, enough traffic, and outcomes slow enough that no test reads them. Before that, the tests’ own controls are the program’s evidence.

What this leaves out

The rest of the product. Feature experimentation on the core product is the same machinery with a different owner, and a product team running it should run it this way, but it is not marketing measurement and it is not specified here. The scope is the acquisition-to-monetization surfaces the growth program is accountable for, whichever team that is in a given company.

Pricing itself. Testing a price is a growth-surface test and this system runs it. Deciding what the price should be is a demand estimation problem with an identification story of its own, and it was scoped for this section once and cut for being a set of methods rather than standing machinery.