One Week of Growth Tests Through the Shipping Rule

Five tests reached a decision this week. None of them needed a meeting, because the argument happened once, when the shipping rule was written, and the rule ran the same way on all five. The meeting keeps the exceptions the rule names in advance, a pricing change with legal exposure or a result several times larger than the surface has ever produced, and this week had none.

Every number is simulated from five invented tests. The shape is the claim.

The week

Week 37 · 5 tests at their stopping rule
Test	Arms	Primary	Guardrails	Verdict
Annual plan as the default togglePricing page · visitor · Fixed horizon, 14 days	47.8k / 48.2kratio p 0.22	+0.30pp+0.07pp to +0.52pp · paid conversion	Refunds breachedWeek-four unreadable	Guardrail breached
Social sign-in above the formSignup flow · visitor · Fixed horizon, 14 days	62.6k / 59.8kratio p < 0.001	+0.46pp−0.00pp to +0.92pp · activation	Refunds passesWeek-four unreadable	Quarantined: assignment ratio
Onboarding checklist, five stepsOnboarding · visitor · Fixed horizon, 14 days	150k / 150kratio p 0.49	+0.10pp−0.23pp to +0.44pp · activation	Refunds passesWeek-four unreadable	Flat: keep the simpler variant
Trial length, 14 to 21 daysTrial-to-paid · account · Fixed horizon, 28 days	1.9k / 1.9kratio p 0.56	−0.96pp−3.10pp to +1.17pp · paid conversion	Refunds unreadableWeek-four unreadable	Inconclusive
Paywall copy, outcome-ledPaywall · visitor · Always-valid, stopped day 9	30.9k / 31.1kratio p 0.53	+0.36pp+0.04pp to +0.69pp · paid conversion	Refunds passesWeek-four unreadable	Ship provisionally, retention pending

The rule runs in order: validity, blocking guardrails, the decision metric against its worthwhile threshold, equivalence, and inconclusive when none of those settled it. Refunds are counted per thousand assigned visitors, not per purchaser, so a variant that changes who buys cannot move the guardrail by changing the denominator. Retention at week four is a delayed outcome, unreadable inside any test shorter than four weeks by construction, and the rule says so rather than treating silence as a pass; a provisional ship keeps its control while it matures.

The row that would have shipped

The social sign-in test shows a +0.46pp lift on activation with an interval that reaches zero only at its edge, and it is quarantined. The treated arm’s assignment records hold 62,573 visitors against 59,834 in control, on a 50/50 split, and the chance of that gap by accident is below one in a thousand. The cause, once found, was that the variant wrote its assignment record when the new form rendered, after a redirect, while control wrote it on arrival; visitors on slow connections who left during the redirect were never recorded as assigned to treatment. That is an assignment defect, not a treatment effect: the randomized population cannot be reconstructed from the records, so no estimator recovers an effect from them. On a dashboard without the check, this test ships on Thursday.

The account-level test whose sample was its account count

The trial-length test randomized 3,800 accounts rather than visitors, because two people at one company cannot see two trial lengths. Those accounts hold about 22,800 seats, and none of them count: the outcome is whether the account converted, one observation per account, so the sample is 3,800 and no seat-level design effect applies on top of it. It could detect 3.1 points on paid conversion against a worthwhile effect of 1.0. The readout says inconclusive, the decision is that a twenty-one-day trial is worth a larger test rather than a drop, and the registry records the account count this one would have needed, which is the number the next test on this surface is planned from.

The one that stopped early

The paywall copy test ran under an always-valid rule and stopped on day nine, when its interval excluded zero: +0.36pp on paid conversion (+0.04pp to +0.69pp). That interval is wider than a fixed-horizon interval on the same data would have been, which is the price of the option to stop, and it is a claim the program can stand behind however many times the dashboard was opened.

The novelty read is on it too. Returning visitors moved +0.92pp and new visitors +0.18pp. That is the pattern of a change existing users notice, and it is the reason the verdict is “ship provisionally, retention pending” rather than “ship”: retention is the delayed outcome a novelty effect fails, and it cannot be read on day nine. The variant ramps with a retained control, the rollback condition stays armed, and the decision is finished at week four.

The pricing win, held

The annual-plan default raised paid conversion by +0.30pp, clear of its worthwhile threshold, and it raised refunds per thousand assigned visitors by 0.61 against a bound of 0.6, with an interval from 0.22 to 1.01 that excludes zero. The guardrail is counted against everyone assigned, not against purchasers, so the extra buyers the variant recruited cannot dilute it. The variant sold annual plans to people who had meant to buy monthly ones. The rule held it, and the decision is not to ship until the variant is fixed rather than to ship and watch, because a blocking guardrail with a stated bound is a decision made in advance and this is the moment it was made for.

The quarter, against the holdback

The registry · 11 weeks into the quarter

Shipped this quarter: 7 tests
Bottom-up forecast, activation: +5.00pp shrunk +3.81pp
What the holdback measured: +1.73pp+0.53pp to +2.93pp

6,000 visitors who saw none of the seven changes, against 114,000 who saw all of them, read on entrants old enough to have matured. The seven shipped effects, all on activation for the same population, sum to +5.00pp as recorded and +3.81pp shrunk toward the surfaces’ history; the holdback says the program added +1.73pp. The residual is the number that grades the shipping rule. What caused it is the next question: the winner’s curse and interactions between changes nobody tested together are the usual suspects, and decay, reach that fell short of the plan, and a proxy that stopped predicting are on the list too.

What this does not establish

That the paywall copy is a win on retention. It is a win on paid conversion with a novelty pattern, shipped provisionally, and retention is read at week four with the control still in place.
That the trial-length change did nothing. It could not be read at this account count, which is a statement about the design, and the registry now holds the account count it needed.
That the seven shipped tests were worth less than recorded. They were worth less in sum than the sum of their records, which is different, and the residual does not say which of its causes was at work.
Anything about a real product. Five tests were invented so the week would need every branch of the rule, and the numbers belong to the script named below.

How this readout was produced Code

Five tests with stated arms, base rates, true effects, true guardrail costs, and a smallest worthwhile effect each. Outcomes are binomial; intervals are normal intervals on differences in proportions at the unit the test randomized, which for the account-level test means the account count with no further design effect. One test carries an assignment ratio skew of four percent in the treated arm, the signature of an assignment record written after a redirect. One runs under an always-valid rule and stops on day nine; its interval is reported as the sequential method would report it. The rule has five branches: quarantine, breach, ship, ship provisionally when only a delayed outcome is unread, flat when the interval sits inside the equivalence region around zero, and inconclusive otherwise. Refunds are counted per thousand assigned visitors. Guardrails have three states, pass, breach, and unreadable, with retention unreadable by construction on any test shorter than its horizon.

Recorded effects are shrunk toward each surface’s history, taken here as a prior of 0.15 points with a spread of 0.4, by the share of each estimate’s variance that is noise. The registry’s quarter is seven earlier shipped tests with stated lifts, and the holdback line is a binomial comparison between the holdback and everyone else with a planted program effect.

Regenerated from seed 20260922 by scripts/growth_readout.py.