Uplift Modeling in Practice
Where to cut the ranking, how much to withhold, and why the offline number is larger than the money. Estimating heterogeneous treatment effects when you have a randomized holdout, and validating the policy prospectively.
A holdout exists. The uplift model is fitted, the deciles separate, and most of the deck is already written. What is left is where to cut the ranking and how much to trust the number at the bottom of the last slide.
Both are easy to get wrong in ways that flip the sign of the program.
The cut is cost over margin
Ranking people by estimated uplift produces an order. It does not produce a policy. A policy needs a stopping rule, and the intuitive one, contact everybody the model scores above zero, is wrong the moment a contact costs money.
The program below sends a retention offer worth $5 to subscribers, and a retained subscriber is worth $85. An offer that raises someone’s probability of staying by one point buys $0.85 of margin for $5. It pays for itself only above cost over margin, which here is 5.9 points.
Nothing in that number is a modeling choice. It falls out of the offer and the margin, and it holds whatever estimator produced the ranking.
Show the data
| Net value, thousands of dollars | Point | 95% CI |
|---|---|---|
| Everyone | -230k | -280k to -181k |
| Highest churn risk | -82k | -101k to -62k |
| Any positive uplift | -20k | -62k to +22k |
| Uplift above break-even | +71k | +52k to +91k |
The offer works on everybody. Sent to all 108,000 subscribers it raises retention by 3.4 points, and it still loses $230,171, because 3.4 points of retention does not cover $5 a head.
Aiming it at the subscribers most likely to lapse is worse per contact than not aiming it at all: 2.4 points against 3.4. That is the propensity trap from Prediction Is Not Persuasion showing up in dollars rather than in a confusion matrix.
The last two rows are the interesting pair. Both use the same uplift estimates and both are correct about who responds. They differ only in where they stop. Cutting at zero contacts 80,824 people and lands on an interval that runs from -$61,658 to $21,526, which establishes nothing. Cutting at break-even contacts 18,299 and clears $51,547 at the bottom of its interval.
There is a harder point underneath the table. The average effect of this offer is 3.2 points, below break-even. Run on everyone, this program should not run at all. The model is not improving a good program. It is answering whether a profitable subset exists inside one that loses money overall, and that is a different question with a different burden of proof.
How much do you have to withhold?
Uplift is a difference between two estimated rates, so its precision is set by whichever arm is smaller. In most programs that arm is the holdout, and the holdout is five percent because five percent is what nobody argues about.
- Value the policy delivers
- Ceiling: perfect knowledge of every effect
Show the data
| Holdout withheld | Value the policy delivers | Ceiling: perfect knowledge of every effect |
|---|---|---|
| 5% | 52.1k | 79.6k |
| 10% | 63.9k | 79.6k |
| 25% | 70.2k | 79.6k |
| 50% | 72.1k | 79.6k |
At a five percent holdout the median cell has 160 untreated subscribers in it, the rank correlation between estimated and true uplift is 0.74, and the resulting policy is worth $52,088. At fifty percent the same cell has 1,571, the correlation is 0.92, and the policy is worth $72,141.
The failure mode is worth naming, because it is not the one people brace for. A noisy uplift model does not turn cautious. It contacts 35,178 people at a five percent holdout and 26,983 at fifty. Noise moves marginal estimates in both directions, and the ones it moves up are the ones that get contacted, so the error is spent rather than avoided.
Returns diminish somewhere around a quarter. Moving from five percent to twenty-five recovers most of the gap; moving from twenty-five to fifty adds $1,983. That is the practical answer to a question usually settled by whoever is most worried about forgone revenue. The cost of a larger holdout appears in a report. The cost of a holdout too small to learn from does not appear anywhere.
The offline number is not the number
The threshold has to come from somewhere. The natural move is to scan candidate thresholds, compute the estimated value of the policy at each, and keep the best one. That is the step where the number stops being an estimate.
The scan maximizes over the same noise the model was fitted on, and the maximum of a noisy surface sits above the surface.
- What the model reports offline
- What the policy delivers
Show the data
| Holdout withheld | What the model reports offline | What the policy delivers |
|---|---|---|
| 5% | 108.5k | 50.9k |
| 10% | 94.3k | 63.9k |
| 25% | 87.0k | 70.5k |
| 50% | 84.5k | 71.8k |
The gap closes as the estimates get more precise, and it never closes entirely. Even at a fifty percent holdout, with this design as sharp as it can be made, the offline figure runs 18 percent high.
This is also the reason an AUUC or a Qini curve computed on the fitting data is not evidence. It is the offline value with a different name and the same defect, and the name sounds like validation.
Validate the policy
Ship the chosen policy to a cohort the model has never seen, withhold a random slice of the people it selects, and measure the difference between them.
Net value of the chosen policy, from a randomized test inside the group it selected
The same policy's offline figure was $86,951, quoted without an interval, because the offline method does not produce one.
- Contacted
- 27,946
- Retention lift
- +9.2pp
- Incremental retentions
- 2,572
- Holdout the model learned from
- 25%
That test asks a different question from the model’s own score. It does not check whether any individual uplift estimate was right, which is not observable and never will be. It asks whether the group the policy picked responds more than a comparable group it did not pick. The money depends on that quantity and on nothing finer.
The interval is wide, roughly
$24,376either side, from a comparison across 27,946 people. That width is the width of the thing being measured. The offline figure was narrower only in the sense that it came with no interval at all.
Choosing an estimator
The estimator matters less than either of the decisions above, close to the reverse of the order most teams work in.
S-learner. One model over both arms with treatment as a feature. Regularization applies to that feature like any other, so a weak treatment signal gets shrunk toward no effect and the model predicts nearly constant uplift. It degrades gracefully and it can be blind to real heterogeneity.
T-learner. A model per arm, differenced. Nothing pulls the effect toward zero, but the two models’ errors do not cancel, and the control model is fitted on the smaller arm. This is what produced every number above, chosen because it makes the holdout-size question visible rather than because it is the best available.
X-learner. Imputes each person’s effect using the other arm’s model, then combines the two estimates weighted by the propensity. Built for the case where the arms are very different sizes, the case most programs are in.
Doubly robust learners and causal forests. Construct a pseudo-outcome that stays unbiased if either the outcome model or the propensity model is right, then regress it on covariates. In a program with a deliberate randomized holdout the propensity is known by construction, the most comfortable position this family is ever in.
With a balanced holdout and enough volume these produce similar rankings and the choice is close to taste. Under a five percent holdout they do not, and the ones that pool information across arms hold up best. If the holdout is small and cannot be enlarged, the estimator becomes the decision worth arguing about.
Context
Best suited for
Programs where the intervention has a real unit cost, runs repeatedly against the same population, and cannot be justified on its average effect alone.
Data requirements
- A randomized holdout large enough that the smaller arm supports a rate estimate inside each region of the feature space
- A known cost per contact and a known margin on the outcome, because the stopping rule is their ratio
- A cohort held back from fitting entirely, for the prospective test
- Features observed before the treatment decision rather than after it
What changes by context
- B2C subscription
- Retention offers are the case where the break-even cut sits highest, because a discount given to someone who was staying costs margin twice over. Expect the profitable subset to be much smaller than the positive-uplift subset.
- Lifecycle-led
- The contact repeats, so a permanent withheld slice costs one cycle of effect and then pays for itself forever. This is the setting where the prospective test is close to free and there is least excuse for reporting an offline number.
- B2B SaaS
- Account-level outcomes rarely support a holdout large enough for this. The move is usually a coarser policy over two or three segments, validated on a longer window, rather than a per-account uplift score nobody can check.
When it breaks down
- Contacts that are effectively free. When cost over margin rounds to zero the stopping rule collapses to "anyone with positive uplift", and most of this lesson stops mattering.
- One-shot campaigns, where there is no second cohort to validate the policy on before it has to be used.
- Populations where the effect is roughly constant. There is no subset to find, and the ranking adds cost without adding value.
- Programs where the holdout cannot exceed a few percent and cannot be pooled across cycles.
What this does not establish
- That a quarter is the right holdout share for your program. It is the point of diminishing returns for this effect size, this base rate, and this many covariate cells. Size it against your own.
- That offline evaluation overstates by any particular amount. The overstatement here runs from 18 to 113 percent depending on the holdout, and the number is a property of how noisy the estimates are, not a constant to correct by.
- That the T-learner is the right estimator. It was chosen to make the holdout-size effect legible. A shrinkage or doubly robust estimator would have softened the five percent result, which is a different lesson.
- That the prospective test settles the question. It measures the policy once, on one cohort, with an interval wide enough to matter. Persuadability moves, and a policy validated last quarter is evidence about last quarter.
- Anything about the size of the gap on a real program. Every figure here comes from one simulated population whose structure was chosen to make the mechanism visible.
What to test next
If a policy is already live and was never validated prospectively, the cheapest useful thing is to start withholding a random slice of the people it selects. No refit, no new model, one flag. Within a cycle there is a defensible estimate of what the policy is worth, and it can be compared against whatever number was used to approve it.
If the holdout is five percent and the argument for enlarging it keeps losing, the argument to make is not statistical. It is that the program’s average effect may sit below break-even, in which case the holdout is not a cost center but the only thing standing between the company and spending money to no effect. The lifecycle uplift example works the comparison through end to end, and Prediction Is Not Persuasion covers why an accurate model does not answer this on its own.
Where these figures came from Code
One simulated retention program of 240,000 subscribers, split into a 132,000 cohort used for fitting and a 108,000 cohort held back entirely. Covariates are discrete, so the uplift estimator is a per-cell difference in retention rates across 40 cells and depends on no modeling library. The generating process gives baseline retention and persuadability different structure, and that is what makes the propensity policy fail; in a population where they coincide it would not. These are not measurements from a real company.
Both headline results are averaged over 240 replications, each re-running the randomization, refitting, and re-choosing the threshold. Bias is a property of repeated experiments, and reporting a single run could not separate optimism from luck. Where one test is shown it is the replication sitting at the median delivered value, never the best one.
The ceiling used as a sanity check is the value of a policy that cuts at break-even using the true individual effects, which exists here only because the effects were generated rather than estimated. On a real program the equivalent check is looser: a policy cannot beat the total effect available in the population, and the total effect is estimable from the same holdout.