Program Lift and Reconciliation
What the program holdout says the lifecycle policy was worth, reconciled against what the tested actions account for on a common value scale, with the residual reported as a residual. The question a per-campaign readout cannot answer.
The decision it encodes
Why did the number move, and how much of it was us?
Retention is up two points this quarter. Forty lifecycle programs ran. Six of them have clean readouts showing a positive effect, and the sum of those six effects is a fifth of the two points.
Somebody now has to explain the other four fifths, and the per-campaign readout cannot do it. It answers whether one campaign worked. It has nothing to say about why the aggregate moved, which is the question being asked in the room.
The decision it encodes
Why did the number move, and how much of it was us?
Both halves matter. Leadership wants the first. The budget conversation depends on the second, and only one component in this system can answer it.
The persistent holdout is the whole answer to the second half
Everything else here decomposes. The persistent holdout from the assignment registry is the only instrument that measures the program against no program, which is the “how much of it was us” question in its exact form.
That makes the top of this reconciliation a subtraction rather than a model. The randomized gap between the treated population and the program holdout estimates the effect of assignment to the lifecycle policy bundle, for this population, over this period, at this outcome horizon. Everything above that line is season, price, product, competition, and whatever else moved, and none of it belongs to marketing no matter how the campaign calendar lines up against it.
Below the line, per-campaign effects cannot be added as they stand. They sit on different eligible populations, different outcome windows, overlapping customers, and sometimes different outcomes. Each is first converted to a common accounting unit: the eligible decision opportunities it covered, times its estimated incremental effect, times the value per incremental outcome from the outcome contract. Those values can be summed and reconciled against the program line. Percentage points cannot.
What the rest of the decomposition is for
Below that line, the question changes from whether the program is worth running to which parts of it are carrying the result.
| Layer | What it rests on |
|---|---|
| Program against no program | The persistent holdout. A measurement, not a model, and the only line here that is. |
| Action against no action or a declared alternative | Per-campaign readouts from automated measurement, converted to incremental value on the common scale, covering only the actions that had a control. |
| Reconciliation residual | Arithmetic. What the holdout says the program did, less what the converted tests account for. It holds untested actions, interactions, cannibalization, and estimation error, in unknown shares. |
| Season, price, product, competition | Whatever sits outside the program contribution. Explanations here are narrative, not estimates. |
Only the first layer is measured. The third is a residual, which means it absorbs every error above it, and the fourth is a story. Reporting all four with the same visual weight is how a decomposition misleads.
Where AI helps, and where it quietly does not
A model is good at the narrative layer. Given the decomposition, the calendar, the price changes, and the release notes, it will write a fluent account of why the quarter went the way it did, in the time it takes to read the prompt. That is a real saving, because writing that account is a slow, annoying job that somebody does every quarter.
It is also the layer with no evidence in it. The danger is not that the model invents a number. It is that the fluency of the explanation gets read as confidence in the decomposition underneath it, and the residual, which is a gap, arrives dressed as a cause.
Maturity, honestly staged
Stage 1: the program line. Persistent holdout against treated, reported on a schedule. One number, measured, and it settles the annual argument about whether the program is worth its cost.
Stage 2: the tested actions beneath it, on the common scale. Per-campaign effects converted to incremental value and stacked under the program line, with the residual shown as a residual.
Stage 3: controls for the obvious. Season, price, and promotion handled explicitly so they stop being absorbed into the residual and blamed on marketing.
Stage 4: a drafted narrative. A model writes the account from the decomposition, and the labeling of what is measured survives into the prose. Last, because it is the layer that most rewards everything underneath being right and most disguises everything underneath being wrong.
Why this is not a mix model Technical
A marketing mix model decomposes an aggregate outcome by regressing it on spend and controls, and it is the right tool where the treatment cannot be withheld from anyone. Lifecycle programs can be withheld from a random slice, which makes the top line of this decomposition a measured difference rather than a fitted coefficient, and that is a far stronger position.
The allocation track covers the mix model case at length, including why fitting a model to observational spend produces a precise answer to a confounded question. The distinction worth holding is that a decomposition anchored on a holdout inherits the holdout’s credibility, and one anchored on a regression inherits the regression’s assumptions.
Context
Best suited for
Programs running enough campaigns that no single readout explains the aggregate, and where somebody is asked quarterly why the top-line metric moved.
Data requirements
- A persistent holdout, without which the top line is a model rather than a measurement
- Per-campaign readouts on a common definition, so they can be stacked at all
- Season, price, and promotion as recorded series rather than recollections
- A stated outcome and window shared with the campaign readouts, or the layers will not add up
What changes by context
- B2C subscription
- Retention moves slowly and for many reasons, so the residual is large and the pressure to explain it is highest. This is the setting where a fluent narrative over a weak decomposition does the most damage.
- Lifecycle-led
- Campaign volume is the reason this component exists. With forty programs running, the sum of the tested effects will always be a fraction of the whole, and the interesting question is whether the untested remainder is worth testing.
- B2C ecommerce
- Promotions dominate and are correlated with campaign timing, so controlling for them explicitly is what stops the decomposition handing promotional lift to lifecycle.
When it breaks down
- No persistent holdout, in which case the top line is a mix model on lifecycle data and should be labeled as one.
- Too few tested campaigns for the second layer to be anything but a residual.
- Aggregate metrics that combine populations with different treatments, where the decomposition is being run at the wrong grain.
- Organizations that will act on the residual. It is the number most likely to be wrong and most likely to be quoted.