Automated Measurement

The campaign ended six weeks ago. It comes up in a planning meeting, someone asks whether it worked, and an analyst spends three days reconstructing the answer: pulling the audience as it was defined then, working out who ended up in which arm, joining delivery events, choosing an outcome window, and picking an estimator.

The answer arrives after the decision it was meant to inform. Worse, the next campaign’s readout gets rebuilt from scratch with slightly different choices, so the two numbers are not comparable and nobody notices, because nobody is holding them side by side.

This component turns that work into a pipeline that has already run.

The decision it encodes

Did this campaign create incremental value, and do we know before someone asks?

The second half is the part that changes behavior. A readout that is available on a schedule gets used in planning. A readout that takes three days gets requested only when someone is already suspicious, which means measurement only ever arrives as an audit.

A readout is a pipeline

Every campaign readout on earth runs the same joins. Eligibility, assignment, delivery, exposure, outcome, guardrails. The work is not intellectually different from one campaign to the next, and the reason it feels different every time is that each of those joins has a choice in it that a bespoke analysis makes silently.

The six joins, and the choice hiding in each

Stage	The choice that has to be fixed in advance
Eligibility	Which audience version, snapshotted when. Live definitions drift mid-test.
Assignment	Which salt and unit, and whether anyone was reassigned after launch.
Delivery	Which stages count as sent, and how retries are treated.
Exposure	What counts as reached, reported as an outcome rather than used as a filter.
Outcome	Which event, and how long a window from the decision rather than from the send.
Guardrails	Which secondary metrics can veto a win, chosen before the result is known.

Fixing the choices in advance is most of what automation buys. The speed is useful. The bigger gain is that the analyst is no longer choosing a window after seeing which window looks better, and does not have to be trusted not to.

What a standing readout has to emit

An effect and an interval is the minimum and it is not sufficient. A readout that reports only a headline hides the two things a reader needs to act.

The effect, as assigned. Everyone who was assigned, including the suppressed, the bounced, and the ignored, with an interval. This is the number that answers whether to run the program.

Exposure rates by arm. Without these, a weak result cannot be told apart from a delivery failure, and those get fixed by different teams.

Balance checks. Pre-period outcomes and a handful of covariates, compared across arms. Imbalance means the assignment or the pipeline did something it should not have, and it is far cheaper to catch here than in the meeting.

Guardrails. Unsubscribes, complaints, support contacts, and margin. A lift that costs you the list is not a win, and the only way that gets caught reliably is if the guardrail is in the readout rather than in someone’s judgment.

A power verdict. What the smallest detectable effect was.

Running on a schedule creates a new problem

Once the readout refreshes weekly, somebody will read it weekly, and will stop the campaign the first week it looks significant. Repeatedly testing a growing sample and stopping on success finds effects that are not there, at a rate far above the one printed next to the interval.

There are two honest fixes and one dishonest one. Fix the readout date in advance and treat everything before it as operational monitoring rather than evidence. Or use a procedure built for repeated looks, which pays for the option to stop early with a wider interval throughout. The dishonest fix is to keep looking and mention only the final number.

Whichever is chosen, the readout should state which one it is, because a reader cannot tell from the interval alone.

Maturity, honestly staged

Stage 1: one readout template, run by hand. Same six joins, same choices, written down. No automation yet, and comparability across campaigns already arrives.

Stage 2: scheduled execution. The readout runs without being asked, and lands somewhere people already look. This is where measurement starts entering planning rather than arriving as an audit.

Stage 3: guardrails and power verdicts in the output. The readout can say a result is inconclusive, and can veto a win on a secondary metric.

Stage 4: variance reduction and sequential procedures. Pre-period covariates narrow the intervals, and repeated looks become legitimate rather than tolerated.

Estimator, variance reduction, and repeated looks Technical

The estimator is a difference in means between arms, with a standard error from the two arm variances. Nothing more elaborate is required, and elaborate estimators on a randomized comparison mostly add ways to be wrong.

Variance reduction is the exception worth taking. Regressing the outcome on its own pre-period value and comparing the residuals removes the part of the variation that has nothing to do with treatment. Where the pre-period correlates strongly with the outcome, which is typical for spend and usage and rare for conversion, this can cut the interval substantially for no additional data and no additional assumption beyond the covariate being measured before assignment.

For repeated looks, the practical options are a fixed analysis date, an alpha spending function that allocates the error budget across a planned number of looks, or an always-valid interval that stays valid under continuous monitoring at the cost of being wider at every point. All three are defensible. Looking weekly against a fixed-sample interval is not, and it is what happens by default the moment a readout becomes a dashboard.

Context

Best suited for

Programs running many campaigns against overlapping populations, where the same six joins are being rebuilt by hand every time and comparability across readouts matters more than bespoke depth on any one.

Data requirements

• Decision and delivery logs joined by a stable decision id
• Assignment recoverable from the unit and the experiment salt rather than from a stored table
• Versioned audience definitions, snapshotted at assignment
• A pre-period window of the same outcome, for balance checks and variance reduction

What changes by context

Lifecycle-led

Campaign volume is the whole argument. Where fifty programs run a quarter, a standing readout is the only way any of them get measured, and the marginal cost of the fifty-first readout is zero.

B2C subscription

Outcome windows are long and the decision is often about retention months out. Reporting several horizons matters more here than anywhere, because a two-week read on a retention offer is almost always misleading.

B2C ecommerce

Purchase timing shifts easily, so a short window flatters anything that creates urgency. The guardrail worth adding is total period revenue rather than campaign revenue, since the failure mode is moving purchases rather than adding them.

When it breaks down

• Campaigns with no holdout, where the pipeline can compute everything except an effect.
• One-off launches with no comparable predecessor, where a bespoke analysis is the right tool.
• Outcomes that arrive later than the planning cycle, so a scheduled readout is always answering a question nobody is still asking.
• Organizations that will not act on an inconclusive verdict, where adding one produces a readout everybody learns to ignore.