Sequential Readouts

The test launched Tuesday. On Thursday the dashboard showed a 6% lift with a p-value of 0.03 and the variant shipped before lunch. By the following Tuesday, had anyone still been looking, the lift was 1% and the interval covered zero.

Nobody did anything unusual. They looked at a test every day and stopped it the first time it crossed the line, which is what every team does until told what that costs.

The decision it encodes

May this test stop now, and what does it say if it does?

A fixed-horizon test answers a question asked once. A test that is looked at daily is asked the question daily, and a procedure that stops the first time the answer is yes will get a yes from nothing eventually. The automated measurement component in lifecycle faces the same problem once a week; here it arrives every morning.

What peeking costs

The arithmetic is unforgiving. A test with no true effect, read once at a pre-set horizon, shows significance at the 5% level five percent of the time. The same test read daily for two weeks and stopped at the first significant day shows it roughly a quarter of the time. The interval reported at that moment is also the interval most favorable to the variant that will ever be seen, because that is when the stopping rule fired.

A test with no effect, stopped the first time it looks significant at 5%

Times looked	Chance of a false win
1	about 5%
2	about 8%
5	about 14%
10	about 19%
20	about 25%

Two rules that work, and the one that does not

A horizon fixed in advance, and not peeked at. Compute the runtime from the detectable effect, run to it, read once. Simple, valid, and hard to hold to on a surface where the dashboard is a browser tab away. A program that chooses this should hide the results until the horizon, not rely on discipline.

An interval that stays valid however often it is looked at. Sequential methods spend the error budget across looks so the total stays at 5%. Group-sequential designs do this at a few pre-planned looks; always-valid methods do it continuously, so a test may be stopped whenever its interval excludes zero and the claim still holds. The price is a wider interval at any one moment than a fixed-horizon test would show, which is the honest price of the option to stop early.

Reading daily and stopping on significance is the one that does not work, and it is the default state of every dashboard.

Checks that run before the lift is read

A readout that starts with the lift has skipped the part that decides whether the lift means anything.

Assignment ratio. A 50/50 test whose assigned arms hold 50,000 and 48,000 visitors is not a 50/50 test, and the imbalance is almost never chance. Bots filtered from one arm, an assignment written after a redirect that drops slow connections, a logging path that fails for one variant: each puts a different population in each arm, and no estimator repairs it. The check is a chi-square on the assigned counts, it costs nothing, and a test that fails it is quarantined until the root cause is classified and the randomized population can be shown to be reconstructable. Most cannot be, and those tests are rerun.

Exposure imbalance is a different thing and is diagnosed rather than condemned. If assignment was recorded before the surface rendered and the arms are balanced on assigned counts, a treatment that loses visitors before render has a lower exposure rate, and that loss is part of its effect. The intention-to-treat read over assigned visitors remains valid; the arms differ in who reached the surface because the variant made them differ. What the imbalance rules out is a triggered read, since the triggered populations are no longer comparable, and what it demands is a look at whether the logging fired at the same moment for both variants.

Assignment integrity. Visitors in both arms, per the assignment record, reported as a rate.

Guardrails. Read against their bounds before the primary, so a variant that wins its metric and breaches a guardrail is seen as a breach first.

The too-good result. A lift several times larger than any past test on the surface is more likely a logging or assignment defect than a discovery, and it earns a check of the pipeline before it earns a celebration.

Maturity, honestly staged

Stage 1: a fixed horizon per test, computed before launch. Even if it is peeked at, a stated horizon changes what “done” means.

Stage 2: assignment ratio and guardrails checked before the lift. The readout has an order, and the order puts validity before the answer.

Stage 3: a sequential method or hidden results. Either the interval is valid under continuous monitoring or the monitoring is prevented. The program stops shipping its own noise.

Stage 4: variance reduction and cohort reads. Pre-experiment covariates narrow the interval where visitors have a history, and new and returning visitors are read separately so a novelty effect is seen rather than shipped.

Always-valid intervals, variance reduction, and what each buys Technical

Group-sequential designs pre-specify a small number of looks and spend the 5% error budget across them by an alpha-spending function, so the test may stop at any planned look with a valid claim. Always-valid methods, built on sequential probability ratio tests, allow a look at any time; the interval at each moment is wider than a fixed-horizon interval on the same data, which is the cost of never being wrong about when you looked. Either is correct. The choice is between a few planned decision points and the freedom to stop whenever, and most growth programs want the second.

Variance reduction with pre-experiment covariates adjusts each visitor’s outcome by its predicted value from their history before the test, which removes the variance the history explains and can narrow the interval by a third or more on returning visitors. It does nothing for visitors with no history, which on a signup surface is everyone, so its value depends on the surface: large on a paywall shown to existing users, small on a signup flow.

The assignment ratio test is a chi-square on assigned arm counts against the intended split. At the traffic these surfaces carry, a ratio that would be chance one time in a thousand is a defect, and the standard practice is to quarantine the test automatically rather than to let a person decide whether the lift is worth keeping. The quarantine lifts only when the defect is classified and the randomized population reconstructed, which is rare; the usual outcome is a rerun with the defect fixed.

Context

Best suited for

Programs where tests are read from a live dashboard by the people who launched them, more than a few run a quarter, and nobody has written down when a test is allowed to stop.

Data requirements

• Arm counts and exposure counts per test, so the sample ratio can be checked
• Outcomes joinable to assignment at the unit the test randomized, with assignment time, so early and late entrants can be distinguished
• Pre-experiment history per visitor where it exists, for variance reduction
• The stated horizon or sequential rule per test, from the registry, so the readout applies it rather than the reader choosing

What changes by context

Product-led

Traffic is high and the temptation to stop early is highest. Always-valid intervals fit the culture; hidden results do not survive it.

B2C subscription

Paywall tests reach returning users with history, so variance reduction earns its keep here more than anywhere. Renewal is the guardrail and it cannot be read inside the test, which is what the holdback is for.

B2C ecommerce

Weekly and promotional rhythms are strong, so the minimum runtime is two full cycles and a test that spans a sale is a test of the sale.

Marketplace

Where the unit is a market rather than a visitor, there are few units and sequential methods have little to work with. The readout looks like a geo test's: one horizon, read once.

When it breaks down

• Surfaces with traffic too low to reach any horizon in a season, where the answer is a bigger change or a coarser metric rather than a cleverer stopping rule.
• Outcomes that arrive after the test can plausibly run, where the primary has to be a validated proxy and the outcome moves to the holdback.
• Organizations that will ship on the first green day regardless, where the useful work is hiding the dashboard rather than improving the statistics on it.