Lifecycle Measurement

A single incrementality study is a project. Somebody scopes it, it runs, it produces a number, and the number goes into a deck. When the question comes back two quarters later, most of that work is done again from the start, because nothing the study built was left standing.

What separates a measurement function from a sequence of studies is the machinery that stays. The diagram above is that machinery for a lifecycle program: what has to exist so a campaign can be measured without a special project, and so the measurement still holds when three other campaigns want the same people next month.

The four stages answer four different questions

Foundation answers who this person is, whether you may contact them, and who is in the audience. Experimentation answers who got what, and whether anyone wrote it down. Measurement answers what happened, how much of it was you, and which estimate governs the decision. Decisioning answers who an action can move, what to do next, and who is allowed to change the rule.

Three layers of randomization, and they answer three questions

The word holdout does three jobs in a lifecycle program, and a system that uses it for all three at once produces numbers that cannot be reconciled.

Three randomizations, kept apart

Layer	How assignment works	The question it answers
Program holdout	A long-tenured slice of customers, at the assignment unit, receives the baseline lifecycle policy instead of any discretionary action for a fixed tenure. Service and legally required messages still go.	Is the lifecycle policy bundle worth running at all?
Campaign control	Among currently eligible customers outside the program holdout, each decision opportunity randomizes the action against no action or a declared alternative.	What is the marginal effect of this action under the current program?
Policy exploration	Inside the ordinary program, feasible actions keep small known probabilities and the probabilities are logged.	Would a different targeting rule perform better than the one running?

A team can be strong at one stage and get nothing back for it. A careful readout on an audience nobody can reproduce describes a population that no longer exists by the time anyone acts on it. A well-specified policy with no logging behind it cannot be compared to the policy that replaces it.

Solid arrows are constraints

Each solid edge is something the component downstream cannot be built without. They are worth reading as failure modes rather than as sequence, because the failures are specific and they are the ones that show up in real programs.

A holdout without an audience definition. You can randomize a list of IDs today. What you cannot do is say who the estimate is about, or rebuild the same population next quarter to check whether the effect held, which is the difference between a result and a finding.

Measurement without exposure logging. The readout has to reconstruct who was treated from send records and opens. That drops everyone the platform failed to deliver to, and it silently converts the question from the effect of assigning the campaign to the effect of receiving it. Those are different estimands with different answers, and the estimand you can identify is the assigned one.

A policy evaluated without logged probabilities. Every action the policy did not take has probability zero in the resulting data, so no alternative policy can be scored against the log. The only available test is to ship the change and wait, which is the position most automated decisioning systems are in.

The loop is the part that compounds

The dashed arrow is the only edge on the diagram that is not a build dependency. It is a runtime relationship: decisions the policy makes become the log the next measurement reads. Remove it and everything left is a project plan that executes once.

Most lifecycle programs run far more campaigns than they learn from, and this edge is the reason. They are rarely short of statistical skill. They are missing the return path, so each campaign is measured, if at all, by a person who went and looked, and the result expires with the send.

One customer, end to end

The components are specified one at a time, and they run as one path. A customer lapses to ninety days and becomes eligible for the winback action. The permission layer confirms marketing email is allowed for her, on this purpose, today. The audience builder places her in the winback risk set at that decision time and freezes it. The registry checks she is not in the program holdout, then assigns her to the campaign’s treatment or control by hashing her id with the campaign’s salt. The policy chooses the offer over no action, at a logged probability, and the decision is written before anything is sent. Delivery succeeds or fails and each stage is logged. Ninety days later her outcome matures and enters the readout as assigned. The campaign’s effect updates the per-state estimates; the candidate policy that would have skipped her is scored on the held-out decisions; and either the rule changes or it does not, with the reason recorded. Nobody ran an analysis, and every number can be traced back along that path.

Where to start when none of it exists

Dependency order is not investment order. The diagram reads left to right because that is what has to exist underneath what. A team building from nothing gets more from starting in the middle.

First, a program holdout and a registry. A slice that receives the baseline policy and no discretionary action, for a stated tenure, and one place that records who is in which test and what each test’s outcome contract is. This settles the annual argument about whether the program is worth its cost, and it needs a hash function rather than a CDP. The compromise is real and worth naming: randomizing on raw IDs before you can define an audience means you have an estimate before you can say who it is about.

Then exposure and decision logging. It makes every campaign after it measurable and does nothing at all for the ones before it, which is the whole argument for doing it early instead of when the first readout needs it.

Then shared definitions: customer state and consent. These are the components that stop two teams building the same audience differently, and they only pay off once more than one team is using them.

Automated measurement, reconciliation, and review after that. They turn the logs into numbers on a schedule, and then into one governing number per decision.

Effects by state, then the policy, with governance around all of it. A next-best-action policy is only as good as the per-state incremental estimates it optimizes against, and those come from pooled randomized decisions rather than from a propensity model. Governance constrains the policy whether a rule engine, a model, or an agent is choosing, and it is drawn last because it constrains something that already runs.

What this leaves out

Paid media. Every component here is person-level, because that is where the hard parts live: identity, consent, contact history, and who wins when two campaigns want the same person. Bought media runs on a different spine and a different unit of randomization, which is why it is a separate system rather than a stage of this one.

Implementation. Each component is written as a specification, because the hard parts are the decisions it encodes rather than the code. Which consent basis applies. Who wins the contact when two campaigns qualify. Whether the organization can serve the audience the statistics require. Those answers are the same whether the stack is dbt and Airflow or a CDP and a scheduler.