Test Design and Traffic

Six tests were live on the signup flow in the same fortnight. Two of them changed the same button. A third ran at a 10% allocation because somebody was nervous, and it was still running eleven weeks later with an interval that had never narrowed. Nobody had decided any of this. Each test had been launched by the team that built it, on the traffic that happened to be free, for as long as it took to look done.

The registry recorded all six, faithfully, after the fact.

The decision it encodes

Which test launches now, on which population, alongside which others, at what allocation, and for how long?

The assignment component makes a test randomized. The metric ladder says what it is read on. This one decides whether it may launch at all, and in what company, which is the decision a program makes forty times a quarter without noticing it is a decision.

Power for the whole rule, before launch

A test is planned against the smallest effect worth acting on, and that number comes from the business rather than from the data: the lift on the decision metric below which the change is not worth its build, its maintenance, or its risk. From it, the metric contract’s variance, and the allocation, the runtime follows, and a test whose runtime exceeds the horizon its surface allows is a test that should not launch as designed.

The power calculation covers the whole shipping rule, not only the primary. A test that must clear its primary and pass three non-inferiority guardrails is four decisions at once, and the chance of passing all four when the variant is in fact fine is lower than any one of them suggests. A guardrail on a rare outcome that cannot be read at the planned sample is written down as unreadable before launch, and the shipping rule treats it as a delayed outcome rather than pretending it will decide anything.

What the launch record settles before the flag turns on

Field	What it decides
Eligibility and trigger	Who enters the test and at what moment, defined before the treatment can affect it and identically in both arms. The analysis population is this, whatever the dashboard later offers.
Smallest worthwhile effect	The lift below which shipping is not worth it. The number the runtime is computed from and the threshold the shipping rule compares against.
Detectable effect and runtime	What the test can see at this allocation inside the horizon the surface allows, for the primary and for each guardrail. A guardrail the plan cannot read is marked unreadable now.
Allocation and ramp	The share of traffic in each arm and the steps to get there: a small first day to catch a broken variant, then the planned split. A test held at a small allocation is a test that will not finish.
Layer and exclusions	Which other tests this one may overlap with and which it may not, so two changes to the same button never meet.
Kill rule	The operational condition that turns the variant off without a readout: an error rate, a latency ceiling, a payment failure. Written down so the first person to notice does not have to decide.
Pre-specified reads	Which segments will be read regardless of the result, so a segment that looks good afterwards cannot be promoted to the finding.

Layers, exclusions, and the tests that must be one test

Independent salts give independent assignments, so two tests on the same surface split the same visitors two different ways and each sees the other’s arms in equal proportion. That is what makes running several at once possible. It does not make it safe. Two tests that change the same element produce four experiences nobody designed; a test on the pricing page and a test on the checkout that follows it interact if the first changes who reaches the second.

The standing answer is layers. Tests that change unrelated things run in separate layers and overlap freely. Tests that touch the same element share a layer and are mutually exclusive: a visitor is in at most one of them, and the layer’s traffic is divided among them. Tests whose interaction is itself the question run as one factorial test, which is the only design that measures the interaction rather than averaging over it.

Ramp, kill, and the checks that run before the first readout

A variant goes to a small share of traffic first, for a day, so a broken build reaches a few hundred visitors rather than half the surface. The ramp is to the planned allocation and it stops there; ramping further while the test runs changes the population over time and confounds the readout with the calendar.

The kill rule is operational and it is not a stopping rule. It turns a variant off on an error rate or a latency ceiling or a payment failure, without a readout, and the test is recorded as killed rather than as a result. A stopping rule decides on evidence; a kill rule decides on damage.

Before any lift is read, the test proves it was measured correctly. An A/A test on a new surface, two identical arms, should show nothing, and one that shows something has found a defect in assignment or logging. Invariant metrics, the ones the variant cannot have moved, should not move. Data freshness, the lag between an event and its appearance in the readout, should be known so a readout is not taken on an arm whose events have not arrived. These are the readout component’s first checks, and they are designed here so the readout has something to check against.

The portfolio

A surface has a fixed amount of traffic and a program has more ideas than traffic. Which tests run is a decision, and the defensible ordering is the same expected-value-of-information logic the performance test calendar uses for channels: what the decision is worth, how uncertain it is, how many visitors the change would reach, what it costs to build and to reverse, and whether the test can read the effect at all. A large idea on a small surface and a small idea on a large surface trade places under that ordering, which is the correction intuition needs most.

Maturity, honestly staged

Stage 1: a runtime and a worthwhile effect per test, before launch. The test has a planned end, and inconclusive can be told from null when it gets there.

Stage 2: layers and exclusions. Same-element tests stop meeting, and the program can say which tests overlapped with which.

Stage 3: ramp and kill rules as configuration. A broken variant reaches hundreds of visitors and turns itself off, and nobody has to decide anything at two in the morning.

Stage 4: the portfolio ordered by decision value. Which test gets the surface is a ranking rather than a queue, and the ranking is written down.

Runtime, guardrail power, and what layering costs Technical

For a conversion metric with base rate p, an even split, and a smallest worthwhile effect of δ in absolute points, the per-arm sample at conventional power is about 2 · (z_α/2 + z_β)² · p(1 − p) / δ², and the runtime is that divided by the arm’s daily eligible traffic, rounded up to whole weekly cycles. Where the unit is above the visitor, multiply the sample by the design effect the unit component computes.

Guardrails are non-inferiority tests against a bound −b, and the power to pass one when the variant is in fact harmless depends on b and the guardrail metric’s variance, not on the primary’s effect. Several guardrails that must all pass multiply their pass probabilities, roughly, so three guardrails at 80% power each pass together about half the time when nothing is wrong. Either the bounds are widened to what the sample can read, or the guardrail is declared unreadable in advance and carried by the holdback.

With k mutually exclusive tests sharing a surface, each gets 1/k of the traffic and its detectable effect scales by √k: 1 test at 1.00×, 2 tests at 1.41×, 3 tests at 1.73×, 4 tests at 2.00×. Orthogonal layers avoid the cost by letting unrelated tests share visitors, which is why the layer assignment is the first thing to settle when a program grows past a few tests at a time.

Context

Best suited for

Programs running more than two or three tests at once on the same surfaces, where launches are decided by whichever team is ready and nobody can say which tests overlapped.

Data requirements

• Eligible traffic per surface per day, to compute runtime from a worthwhile effect
• Metric contracts from the ladder, for the variance the power calculation needs
• A flag tool that supports layers or namespaces, or a convention that enforces exclusivity by hand
• Operational metrics at the variant level, so a kill rule has something to fire on

What changes by context

Product-led

Traffic is high and the constraint is collisions rather than power. Layers are the whole component here, and the portfolio question is which of many worthwhile tests gets the signup flow this month.

B2C subscription

Paywall and pricing tests cannot overlap with each other and often cannot overlap with promotions, so the layer is small and the calendar matters more than the traffic.

B2C ecommerce

Seasonality sets the runtime floor and promotions set exclusions: a test that spans a sale is a test of the sale, so the launch record carries the promotional calendar.

B2B SaaS

At the account level there is rarely traffic for more than one test on a surface at a time, so the portfolio decision is which single test runs this quarter, and the runtime is measured in months.

When it breaks down

• Programs running one test at a time, where layers are ceremony and the useful parts are the worthwhile effect and the kill rule.
• Surfaces where the flag tool cannot express exclusivity, in which case the convention has to live in a calendar somebody maintains by hand.
• Ideas whose worthwhile effect nobody will state, where the runtime cannot be computed and the test will end when someone is tired of it.