The Metric Ladder and Guardrails

Over a year the signup flow was tested eleven times and signups rose 31%. Over the same year, the share of signups who reached the first meaningful action fell from 44% to 33%. Every one of the eleven tests was read on signups, every one won, and the program’s most confident metric had spent the year recruiting people who did not want the product.

Nothing was wrong with any test. The metric was wrong for all of them.

The decision it encodes

Which metric decides this test, which metrics can veto it, and how was each proxy validated?

Three decisions, each made before the test launches, each recorded so the readout applies them rather than the meeting reinterpreting them. The decisioning track works through why the deepest metric is usually the right target and rarely the right test metric; this component is that argument turned into standing rules.

The ladder

The growth surface has a funnel and the funnel is a ladder of metrics: visit, signup, activation, paid, retained, and under all of them the value each eligible visitor was worth. Each rung is closer to what the business wants and further from what a two-week test can measure. Signups are abundant and arrive in the session. Retained value is the thing that matters and arrives in a quarter or a year.

The decision metric for a test is the deepest rung the test can read with power in the time it is allowed to run, chosen before launch from a standing policy for that surface and kind of change, and the answer is usually activation: a defined first action that predicts retention and arrives within days. Signups are a diagnostic rather than a decision metric, because a change can raise them by lowering the bar, and the year above is what that looks like. On pricing and paywall surfaces the decision metric is value per eligible visitor at a fixed horizon, revenue or contribution, because paid conversion alone rewards the discount that sells more of less.

One surface’s ladder, and what each rung is for

Rung	Arrives	Role in a test
Signup	In the session	Diagnostic. Tells you whether the variant was seen and acted on; cannot decide a test on its own.
Activation	Within days	Decision metric for most surfaces. Defined once per product, validated against retention, and not redefined per test.
Paid conversion	Days to weeks	Diagnostic on pricing and paywall surfaces: it explains why value moved. A lower price raises it while lowering what each visitor was worth.
Value per eligible visitor	Weeks, at a fixed horizon	Decision metric for pricing and paywall surfaces: revenue or contribution per randomized visitor at D30 or D90, refunds netted. The rung the others stand in for.
Retention and retained value	A quarter or more	Delayed outcome. Read inside a test only when the horizon allows, and otherwise from the holdback. Too slow to decide a test; too important to leave out.
Refunds, support contacts, latency	Days	Blocking guardrails, counted per randomized visitor rather than per purchaser. A variant that wins the decision metric and breaches one has not won.
Arm counts, exposure rates, invariant metrics	Immediately	Quality checks. They say whether the test was implemented and measured correctly, and nothing about whether the change worked.

Four roles, and a contract for each

A metric in a test plays one of four roles, and the role is written down. The decision metric says whether the change created enough value to ship. A blocking guardrail matures inside the test horizon and can stop a ship on its own. A delayed outcome is what the program is for and cannot be read before the decision; it is carried by the holdback and by a provisional ship that keeps its control. A diagnostic explains how the effect happened and cannot declare a win, and a quality check says whether the test was measured correctly at all.

Every metric in the first three roles carries a contract: the unit it is counted at, the eligible population, the numerator and the denominator, the observation window and when it counts as mature, the direction that is better, what happens to missing data and outliers, and, for a guardrail, the non-inferiority bound. The denominator is where contracts earn their keep. Refunds among purchasers and retention among converters are useful diagnostics and poor guardrails, because a variant that changes who buys changes the denominator; refunds per randomized visitor and retained subscribers per randomized visitor keep the comparison the randomization made.

A guardrail does not have to improve; it has to not degrade beyond its bound. The list is short, the bounds are written before launch, and a variant that wins its decision metric while breaching a guardrail does not ship, however the meeting feels about it. Refunds and support contacts catch the pricing test that converted people who did not understand what they bought. Page latency catches the variant that won because it was tested on fast connections. Retention is on every list as the delayed outcome, read inside a test only when the horizon allows and otherwise from the holdback. A guardrail the planned sample cannot read is declared unreadable before launch rather than discovered unreadable after, and several guardrails that must all pass are powered as one decision, since three bounds at 80% power each pass together about half the time when nothing is wrong.

How a proxy earns its place

Activation stands in for retention. That is a claim, and it is testable: across the tests that have already run, did lifts on activation predict lifts on retention read later? A proxy whose test effects predict the outcome’s out of sample, with calibrated uncertainty, within the kind of change being tested, earns provisional standing to decide tests of that kind. One whose effects do not is a metric the program is optimizing for its own sake, and the year at the top of this page is what that produces. The qualification about the kind of change matters: activation can predict retention for onboarding simplifications and fail to predict it for discounts, because a discount recruits a different person.

The check is cheap once the registry exists, because every past test carries both numbers. It is also the reason the holdback exists: a proxy can predict retention across tests and still fail to predict it in aggregate, and only a standing comparison catches that.

Maturity, honestly staged

Stage 1: one decision metric per surface and kind of change, written down. The team stops choosing the metric after seeing the results, which is most of the improvement available.

Stage 2: guardrails with bounds and contracts. Refunds per assigned visitor and one operational metric, each with a denominator, a window, and a degradation the variant may not exceed, tested for non-inferiority rather than glanced at, with retention named as the delayed outcome.

Stage 3: proxies validated against the outcome across past tests. Activation has to have predicted retention before it is allowed to decide a test, and the validation is repeated as tests accumulate.

Stage 4: the ladder maintained from the holdback. A proxy that predicted the outcome test by test and not in aggregate is demoted, and the ladder changes because the evidence did.

Non-inferiority bounds and proxy validation Technical

A guardrail with bound δ passes when the confidence interval for the treatment effect lies entirely above −δ. That is a one-sided non-inferiority test, and its power depends on δ rather than on any expected improvement, so a guardrail on a slow, noisy metric like retention often cannot pass inside a two-week test. The honest treatment is to state the bound, report the interval against it, and let the holdback carry the retention question when the test cannot.

Proxy validation across tests is a regression of outcome lift on proxy lift, one point per past experiment. The slope is not the test; a one-point lift on activation might reliably predict four tenths of a point on retention and be an excellent proxy. What matters is whether the relationship predicts held-out experiments’ outcome lifts, with calibrated error, with the sign right, and stably across the kinds of change and the cohorts the program runs. Both axes are estimated with noise, often from the same experiment, so a precision-weighted line understates the uncertainty and a hierarchical measurement-error model is the defensible version once there are enough tests to fit one. Predictive alignment is evidence for a proxy and not proof of causal surrogacy, which rests on assumptions about the pathway that no regression checks; the holdback’s aggregate read is the closest the program gets to testing those. The idea generalizes to a surrogate index built from several early metrics, which can predict a long-run outcome better than any single proxy, at the cost of being harder to explain in the meeting.

Context

Best suited for

Programs running more than a few tests a quarter on the same surfaces, where the metric that decides a test is currently chosen per test and retention is read, if at all, months after the variant shipped.

Data requirements

• A defined activation event per product, stable across tests, from the product analytics
• Retention at a fixed horizon, joinable to the assignment record
• Refunds, support contacts, and latency per visitor or per account, at the unit the test randomized
• Past test results with both proxy and outcome lifts, from the registry, to validate proxies against

What changes by context

Product-led

Activation is the whole game and the definition is contested. Settling it once, from the actions that predict retention, is the highest-value decision in this component.

B2C subscription

Paid conversion is the primary on paywall surfaces and refunds are the guardrail that catches a discount that converted people who churn in month two. Read retention at the first renewal, whatever the test length.

B2B SaaS

Activation is a team event rather than a user one, and the outcome is an expansion or a renewal a year out. The ladder is longer, the proxies are weaker, and the holdback carries more of the weight.

B2C ecommerce

Purchase is fast enough to be the primary on most surfaces, and the guardrail is return rate and margin, since a variant can raise orders by selling the wrong thing.

When it breaks down

• Products with no activation event that predicts anything, where every test is read on signups and the ladder has one rung.
• Programs with too few past tests to validate a proxy, where the outcome has to stay a guardrail and most tests will be inconclusive on it.
• Surfaces where the business wants a metric the test cannot read, such as lifetime value on a two-week pricing test, where the answer is a proxy plus the holdback and not a longer dashboard.