Treatment Effects and Policy Evaluation

The winback campaign’s readout says +1.4 points on everyone assigned. The next-best-action policy wants to know whether this particular customer, lapsed ninety days with two prior purchases and one unanswered message, is worth the offer. The readout does not say. Nothing in the program says, so the policy scores her with a propensity model, ranks her near the top because she is likely to come back anyway, and spends the offer on a return that was already coming.

The average effect was measured well. The decision needed a different number.

The decision it encodes

For this customer state, what is the expected incremental value of each action, and would a candidate policy beat the one running?

Automated measurement answers whether a campaign created value on average. Next best action needs, for each customer and each candidate action, the incremental value of taking it. Those are different estimation problems, and a program that skips from one to the other fills the gap with a propensity score and calls it uplift.

Propensity is who will convert. Uplift is who can be moved

A customer very likely to renew is not a customer whose renewal a message can change. Propensity models rank the first; a policy needs the second; and the two rankings disagree most exactly where the money is, because the customers most likely to convert are the ones a contact adds least to. Uplift modeling in practice works through the mechanics and the failure. Here the point is architectural: the uplift estimate is a product of the measurement layer, built from randomized decisions, and it does not come from the same model that predicts behavior.

What it is built from, and what it may read

The raw material is the randomized decision opportunities the program has already logged: every customer × decision time where the action was assigned by a rule with a known probability, with the outcome that followed. Campaigns whose treatment, outcome, and horizon are comparable pool; ones that are not stay separate, because an uplift estimate averaged over a discount and a newsletter describes neither.

The features are the customer’s state as of the decision, from the state model at that timestamp, never as of now. A feature computed after the decision has already seen the outcome, and a model trained on it will validate beautifully and fail on arrival.

What the component emits, per customer state and action

Output	What it is for
Incremental effect by state	The estimated lift of each action for customers like this one, on the primary outcome at its horizon, with an interval. The number the policy ranks on.
Support	How many randomized decisions like this one the estimate rests on. A segment with forty observations gets shrunk toward the average, and the policy is told so.
Incremental value	The effect converted through the outcome contract: value per incremental outcome, less the action’s cost and any discount leakage, over the declared horizon.
Candidate policy value	What a proposed targeting rule would have earned on held-out randomized decisions, with an interval, against the rule that ran.
Overlap and effective sample	Where the logged policy gave the candidate’s choices enough probability to evaluate them, and how much information survives the reweighting.
Version and training window	Which decisions and which state definitions produced these estimates, so a policy can be traced to the evidence it stood on.

A policy is evaluated before it controls anything

An uplift model produces a ranking. A ranking with a threshold and constraints is a policy, and the question that matters is not whether the model fits but whether the policy earns more than the one running. That is answered on randomized decisions the model never saw: hold out a share of logged decision opportunities, apply the candidate rule to them, and estimate what it would have earned by reweighting the outcomes by the ratio of the candidate’s action probability to the logged one.

The rule that follows is simple to state and expensive to skip: a policy controls meaningful volume only after its out-of-sample value has been estimated and has beaten the incumbent by more than its interval. A policy that cannot be evaluated on the log gets a small live arm first, under the registry’s policy-versus-policy design, and earns volume from that.

Maturity, honestly staged

Stage 1: average effects from the readouts, applied to everyone. The policy ranks by cost and margin on a common effect. Crude, and already incremental rather than propensity-based.

Stage 2: effects by a few declared states. Lapsed against active, tenure bands, prior contact count: the segments the program already argues about, estimated from pooled randomized decisions with shrinkage toward the average.

Stage 3: a fitted heterogeneous-effect model with support reported. The estimate varies continuously with state, every prediction carries how much evidence it rests on, and small segments are pulled toward the mean by rule.

Stage 4: candidate policies scored on held-out randomized decisions before they ship. The value of a rule, with its interval and its effective sample, is a standing output, and no policy takes volume without one.

Estimating heterogeneous effects and scoring a policy Technical

The estimand per state x and action a is the conditional average treatment effect τ(x, a) = E[Y | X = x, A = a] − E[Y | X = x, A = none], on randomized decision opportunities. The workable estimators are the ones the uplift literature settled on: separate outcome models per arm differenced (a T-learner), a single model with treatment as a feature (an S-learner), an orthogonalized learner that models the outcome and the propensity first and fits the effect to the residuals (the R- or DR-learner), and causal forests, which fit τ directly with valid intervals. On randomized data with a known assignment probability the propensity step is trivial, which is the whole reason this component sits downstream of randomization rather than of observational logs.

Shrinkage is not optional. Each state’s estimate is pulled toward the program average in proportion to its noise, the same empirical-Bayes move the growth registry makes for shipped tests, and a minimum-support rule refuses to let a state with fewer decisions than the rule allows carry its own estimate at all.

Policy value is estimated by inverse-propensity weighting on held-out decisions: each logged outcome is weighted by the candidate’s probability of the logged action over the logging policy’s, and the doubly robust form adds an outcome model so the estimate is consistent if either piece is right. The effective sample size after weighting is (Σw)² / Σw², and it is reported beside the value, because a candidate far from the incumbent can have a plausible point estimate resting on a handful of decisions. Where it is small, the answer is a live policy-versus-policy arm, not a tighter model.

Context

Best suited for

Programs whose next-best-action policy currently ranks on a propensity score, or on a single average effect applied to everyone, and that have enough randomized decisions logged to estimate anything finer.

Data requirements

• Randomized decision opportunities with the assignment probability, the action, and the outcome at its horizon, from the decision log
• Customer state as of each decision, from the point-in-time state model
• The outcome contract: value per incremental outcome, action cost, and horizon, so effects become values
• A held-out share of logged decisions the model never trains on, for policy evaluation

What changes by context

B2C subscription

Retention effects are heterogeneous by tenure and by how the customer arrived, and the discount actions have costs large enough that the value conversion changes the ranking. This is where an incremental policy earns the most over a propensity one.

Lifecycle-led

Decision volume is high enough to estimate effects by state within a quarter, and high enough that a policy shipped without evaluation does a quarter of damage before anyone reads it.

B2C ecommerce

Purchase timing makes the horizon load-bearing: an effect estimated at two weeks rewards pull-forward, and the value conversion has to net out purchases that would have happened the following month.

Product-led

Usage state is rich and changes daily, so the as-of rule on features is the most common way this component goes wrong. A model reading current usage has seen the outcome.

When it breaks down

• Programs with few randomized decisions, where the honest output is a shrunk average effect and a policy of the simplest kind.
• Logged policies that never explored, so no candidate that differs from the incumbent has support and nothing can be evaluated off the log.
• Outcomes whose value nobody will state, since an effect in points cannot be ranked against a cost in dollars.
• Organizations that will ship the model on its validation score, where the policy-evaluation output exists and governs nothing.