Shipping Rules and the Registry

Two tests read out the same week. One showed a 3% lift with an interval from 1% to 5% and did not ship, because the product lead preferred the control. The other showed a 2% lift with an interval from −1% to 5% and shipped, because the designer had spent a month on it. Both decisions were defensible in the room. Neither was a rule, and by the end of the quarter the program had shipped whatever its most persuasive person wanted, with confidence intervals as decoration.

The decision it encodes

What ships, what dies, what an inconclusive result does, and where the answer is kept?

The readout says what the data showed. This component says what happens next, and it says it before the readout exists, so the same result produces the same decision whoever is in the room. The evidence review in lifecycle does this for a weekly queue; here the volume is higher and the rule has to run without a meeting.

The rule

Five outcomes, and each one has a decision attached in advance. The threshold the decision metric is held to is the smallest worthwhile effect stated before launch, not zero: a lift that excludes zero at a large sample and falls short of what the change costs to build, maintain, and support is a finding and not a ship.

The decision metric clears the worthwhile threshold and every blocking guardrail holds. Ship. The effect recorded in the registry is the interval, not the point, and the point is shrunk toward what past tests on this surface have found, because the tests that win are the tests whose noise fell in their favor.

The decision metric clears, the blocking guardrails hold, and a delayed outcome has not matured. Ship provisionally. The variant ramps while a randomized control is retained, retention or renewal keeps maturing in both arms, the rollback condition stays armed, and the final decision is taken at the horizon the delayed outcome needs. This is the branch a fast loop runs most often, and naming it is what keeps “ship and watch” from meaning “ship”.

A blocking guardrail breaches, or the decision metric is harmful. Do not ship, whatever else moved. A pricing variant that raised paid conversion and raised refunds per assigned visitor past the bound is a variant that sold something people did not want, and the decision metric is not allowed to outvote that.

Flat, inside the equivalence region. The interval sits within the band around zero the program said was not worth acting on, in either direction. The variant did nothing worth shipping. Ship the simpler, cheaper, or safer one, which is often the control. A flat result is a finding about the surface, and it goes in the registry as one.

Inconclusive. The test could not distinguish a worthwhile effect from nothing from harm. What happens next depends on what the decision is worth: a low-value question is recorded and dropped; a test whose exposure or measurement was weak is redesigned; a high-value question gets a larger confirmatory test; and a test is extended only under a sequential or sample-size rule that was written before launch, because extending it because the result was not the one wanted is peeking with extra steps. The registry marks it, and the surface’s next test is planned with the runtime this one needed.

The registry

Every test gets one record, opened before launch and closed by the rule. What was asked, on which surface, at which unit, by what mechanism it was expected to work and what would have shown it did not, with which decision metric and guardrails and bounds and worthwhile effect, with what horizon or stopping rule; what was found, as intervals; what was decided, by which branch of the rule; what the shrunk effect is; and what was learned that the next test can use.

The registry is what makes the loop compound rather than repeat, in two ways. Statistically, past effects on a surface are the prior for the next test’s power calculation, so the tenth test on the pricing page is planned from nine results rather than a guess, and the registry is where the metric ladder gets the pairs of proxy and outcome lifts it needs to keep validating proxies. Mechanically, a flat result that says “shorter forms raised form starts and did not move activation, so the friction is after account creation” is worth more than a recorded zero, because it says where the next hypothesis lives. A registry that carries only effect sizes accumulates precision; one that carries mechanisms accumulates understanding.

What a registry entry carries, and when it is written

Field	Written	What it decides
Surface, unit, hypothesis, mechanism	Before launch	What is being tested, on whom, why anyone expected it to work, and which intermediate metric would move if the mechanism is right, so a flat result can say which part of the story failed.
Decision metric, guardrails, bounds, worthwhile effect	Before launch	What decides, what can veto, and the smallest lift worth shipping, fixed so none of them can be chosen after the fact.
Horizon or stopping rule, layer, and the detectable effect it was planned for	Before launch	When the test may stop, which tests it may overlap with, and how big an effect it could see, so inconclusive can be told from null.
Pre-specified segment reads	Before launch	Which segments will be read whatever the result, so a segment that looks good afterwards cannot be promoted to the finding.
Results, as intervals, on every registered metric	At the readout	What was found, including the guardrails, including the ones that did not matter.
Decision and the branch of the rule that produced it	At the readout	What happened and why, in a form the next person can audit.
Shrunk effect	At the readout	The estimate the program carries forward, pulled toward the surface’s history by how noisy this test was.
What was learned, and the next test it implies	At the readout	Beyond ship or not: what the mechanism metrics said, whether the finding is believed to generalize past this surface, and whether it needs replicating before anyone builds on it.

Why the rule has to run without a meeting

Forty tests a quarter is forty decisions, and a program that makes each one in a room makes them differently, slowly, and in favor of whoever argued best. The rule is the same rule every time, so the meeting is for the cases the rule did not anticipate, which are few, and for deciding what to test next, which is the only decision that needs judgment.

That is also what lets the team that owns the surface ship without waiting. The rule was set once with a data scientist in the room; applying it needs nobody in the room at all.

Maturity, honestly staged

Stage 1: the five-branch rule written down. Ship, ship provisionally, do not ship, ship the simpler, inconclusive with its next step. The meeting stops deciding the ordinary cases, and keeps the exceptions: a pricing change with legal exposure, an irreversible migration, a result several times larger than anything the surface has produced, each of which earns a replication or a review before it ships.

Stage 2: pre-registration. Primary, guardrails, bounds, and horizon written before launch, in a record that cannot be edited afterwards.

Stage 3: shrinkage on recorded effects. The number carried forward is pulled toward the surface’s history, and the sum of the year’s wins stops being a number nobody believes.

Stage 4: priors from the registry. Power calculations start from what past tests on the surface found, and the program knows before launch which ideas are too small to detect.

Shrinkage, the winner's curse, and priors from past tests Technical

Selecting tests to ship on significance means the recorded effects are biased upward, by more for noisier tests, since a noisy test needs a lucky draw to clear the threshold. The correction is empirical Bayes: treat the distribution of true effects on a surface as estimated from its past tests, and shrink each new estimate toward that distribution’s mean in proportion to how much of its variance is noise. A test with a wide interval is shrunk hard; a precise one barely moves. The shrunk value is what the registry carries and what the sum of wins is built from.

The same distribution is the prior for planning. If past effects on a surface have been a few percent with most inside two points of zero, a new test planned to detect ten percent is planned to be inconclusive, and the registry says so before the launch rather than after. The uplift lesson makes the parallel point about scores that look good offline: an estimate selected for being large is smaller than it looks.

Interactions between concurrent tests are usually small and occasionally not. Recording concurrency in the registry does not fix them; it makes them findable when the holdback and the sum of wins disagree.

Context

Best suited for

Programs running enough tests that the decisions cannot all be made in a meeting, where the metric or the decision is currently chosen after the results are seen, and where nobody can produce last year's tests with their results.

Data requirements

• A registry that is written before launch and read-only after the readout
• Readouts in the registry’s terms: intervals on every registered metric, plus the sample ratio and guardrail checks
• Past effects per surface, to compute shrinkage and to plan power
• Concurrency: which tests ran on the same visitors in the same period

What changes by context

Product-led

The volume is highest and the rule matters most. Flat results are common and the "ship the simpler" branch does most of the work, which is unglamorous and correct.

B2C subscription

Pricing tests have the most persuasive people in the room and the strongest reasons to ship a marginal win. The refund guardrail is the branch that saves the quarter.

B2B SaaS

Few tests, at the account level, with wide intervals, so most readouts are inconclusive and the rule’s fourth branch runs most often. The registry’s value is in the power planning: knowing what cannot be detected before spending a quarter on it.

B2C ecommerce

Seasonality means a winner from October may not be a winner in February. The registry records when a test ran, and a shipped effect from a peak month is shrunk against a history that includes the troughs.

When it breaks down

• Programs where shipping is decided elsewhere and the test is confirmation, in which case the rule governs nothing and the registry records decisions it did not make.
• Surfaces with a handful of past tests, where shrinkage has nothing to shrink toward and the prior is a guess dressed as history.
• Teams that edit the registry after the readout, which turns pre-registration into a notebook and the rule into a description of what happened.