Prediction Is Not Persuasion
A model that knows who will convert does not know who you can convince. The two questions have different answers, and only one of them can be checked against data you already have.
A churn model reaches 0.84 AUC. The retention team points a save offer at the top decile. Six months later churn is unchanged, and nobody can say why.
Nothing was wrong with the model. It was asked the wrong question.
The two questions
Prediction answers: who is likely to do this? Churn, convert, activate, buy again. A model learns it from history, and you can check how well it did as soon as the outcome arrives.
Persuasion answers: whose behavior would change if we acted? That is a difference between two worlds, one of which did not happen.
Every targeting decision is a persuasion question wearing prediction’s clothes. You are not trying to find people who will churn. You are trying to find people who will churn and would stay if you intervened. Those are different groups, and the second is usually much smaller.
Four kinds of customer
Sort everyone by what happens with and without the intervention, and there are only four possibilities.
| Converts if contacted | Does not convert if contacted | |
|---|---|---|
| Converts if left alone | Sure thing | Sleeping dog |
| Does not convert if left alone | Persuadable | Lost cause |
Sure things convert either way. Contacting them costs money and changes nothing.
Persuadables convert only because you acted. They are the entire return on the campaign.
Lost causes never convert. Contacting them is wasted but harmless.
Sleeping dogs would have converted, and your intervention talked them out of it. Another five emails, an unsubscribe, and a subscription that never happens.
A prediction model scores people by probability of converting. That ranking puts sure things at the top, because they are the most likely converters in the data. Sleeping dogs rank high too, for the same reason. Persuadables sit in the middle of the ranking, where nobody is looking.
What it costs
This is not a theoretical concern. The lifecycle uplift example runs both policies against the same randomized holdout, on the same population, with the same contact budget.
- Ranked by uplift
- Ranked by propensity
Show the data
| Period | Ranked by uplift | Ranked by propensity |
|---|---|---|
| 0% | 0.0 | 0.0 |
| 569.8 | -58.4 | |
| 10% | 928.0 | 7.0 |
| 1,388.9 | 33.2 | |
| 20% | 1,713.7 | 102.2 |
| 2,047.9 | 200.6 | |
| 30% | 2,246.2 | 353.1 |
| 2,306.7 | 632.8 | |
| 40% | 2,346.0 | 995.7 |
| 2,423.9 | 1,399.3 | |
| 50% | 2,453.5 | 1,734.7 |
| 2,466.3 | 2,199.3 | |
| 60% | 2,435.3 | 2,312.4 |
| 2,435.0 | 2,339.1 | |
| 70% | 2,460.9 | 2,377.4 |
| 2,556.0 | 2,408.4 | |
| 80% | 2,554.4 | 2,447.7 |
| 2,604.5 | 2,490.5 | |
| 90% | 2,594.9 | 2,502.7 |
| 2,586.8 | 2,464.5 | |
| 100% | 2,487.3 | 2,487.3 |
Same budget of 19,287 contacts. Ranking by predicted conversion probability produced 996 incremental conversions. Ranking by estimated uplift produced 2,346, which is
2.4times as many from the same spend.
The more uncomfortable number is the comparison with doing no targeting at all. Contacting everyone delivered 51.6 incremental conversions per thousand contacts. The propensity policy delivered 51.6. A model with genuine predictive accuracy produced a policy no better per contact than not targeting.
Why you cannot fix this with a better model
The instinct is to improve the model. More features, better architecture, careful calibration. None of it helps, because the problem is not accuracy.
To know who is persuadable you need to observe what happens to comparable people you did not contact. That information does not exist in a dataset where everyone was contacted. It is not hidden in the features waiting to be extracted. It was never recorded.
That is the real cost of never holding anything out. It is not just that you cannot measure the campaign afterwards. It is that you can never build a model of who to target, because the training data for that model has to be created deliberately and cannot be recovered later.
What targeting persuadability looks like
Once a holdout exists, the machinery is not exotic. Fit one model on the treated group and one on the control group, and take the difference between their predictions. That difference is an estimate of the effect for that person, and you rank by it.
The awkward part is validation. You never observe any individual’s uplift, so there is no per-person error to compute. What you can do is sort the validation set by predicted uplift, cut it into groups, and measure the actual effect in each group from the randomization.
- Predicted
- Observed
Show the data
| Period | Predicted | Observed |
|---|---|---|
| D1 | 18.7pp | 19.2pp |
| D2 | 16.6pp | 16.3pp |
| D3 | 12.4pp | 10.7pp |
| D4 | 4.0pp | 2.1pp |
| D5 | 2.2pp | 2.2pp |
| D6 | 0.8pp | -0.2pp |
| D7 | 0.6pp | 0.1pp |
| D8 | 0.5pp | 2.0pp |
| D9 | -0.1pp | 0.6pp |
| D10 | -4.0pp | -1.9pp |
A ranking that separates the top from the bottom is usable. One where the deciles are flat is not, however good the underlying models looked in isolation.
Note the bottom decile. Both the prediction and the measurement are negative, which is the model identifying people the campaign loses. A propensity model cannot express that at all. Negative persuadability has no representation in a framework where the only output is a probability of converting.
Where prediction is still the right tool
None of this makes predictive models useless, and swapping every score for an uplift model is its own mistake.
Prediction is the right tool when the decision does not involve intervening on the person being scored:
- Forecasting. How many conversions should we staff for next month.
- Prioritization under a hard capacity limit where every contact is going to happen anyway and the only question is order.
- Eligibility and risk. Whether someone qualifies, not whether they can be moved.
- As an input to an uplift model. Baseline propensity is one of the two quantities the difference is taken between.
The distinction is whether you are asking what will happen or what you can change. Only the second one needs a counterfactual.
Context
Best suited for
Any decision about who receives an intervention, where the intervention has a cost and the audience is larger than the budget.
Data requirements
- A randomized holdout, past or planned. This is the binding requirement
- Features observed before the treatment decision rather than after it
- Enough volume that a tenth of the validation set still supports a rate estimate
- An outcome that arrives inside a window the business will wait for
What changes by context
- B2C subscription
- Churn and winback are where sleeping dogs bite hardest. A save offer can remind a quiet subscriber that they are paying for something, and a propensity model ranks those people at the top.
- B2B SaaS
- Account-level outcomes cut the sample size enough that deciles often will not hold. Coarser groups, or a holdout run over several quarters, are usually the options left.
- Lifecycle-led
- The natural home for this, because the contact is repeatable and a permanent holdout costs almost nothing. Programs here have the least excuse for not knowing their uplift.
- Product-led
- The intervention is often a product change rather than a message, which makes it harder to withhold cleanly and easier to leak between users. Feature flags are the usual mechanism and they need the same randomization discipline.
When it breaks down
- No holdout and no willingness to create one. Nothing here works on observational data.
- Effects small relative to noise, where every decile looks the same and the ranking carries no information.
- One-shot campaigns, since the ranking cannot be validated before it has to be used.
- Very low base rates, where a tenth of the validation set contains too few conversions to estimate a rate.
What this does not establish
- That uplift modeling always beats propensity targeting. It establishes that they answer different questions. Where effects are homogeneous, ranking by uplift adds nothing.
- That the four-way segmentation is observable. Nobody is labeled a persuadable in the data. It is a way of thinking about the effect, not a variable you can select on.
- That a validated ranking stays valid. Persuadability moves with price, competition, and the product, and usually faster than baseline conversion does.
- Anything about how large an uplift model's advantage would be on your program. The gap in the figures above is a property of one simulated population.
What to test next
If you have no holdout anywhere, start one. A permanent five percent withheld from a single lifecycle program costs very little and is the only way to find out whether any of this applies to you.
If you have a holdout and a propensity model, the cheapest useful thing is to sort the holdout by that model’s score and measure the actual effect by decile. If the effect is flat across the ranking, your targeting is not doing what the accuracy metric suggested, and you now know it for the price of an afternoon.
The lifecycle uplift example carries that comparison through end to end, including what each policy is worth once contact costs are accounted for.
Where these figures came from Code
Both charts come from the lifecycle uplift worked example: one simulated trial-to-paid nurture program, a randomized 50/50 holdout, and policies scored from the randomization on a validation cohort the models never saw. They are not measurements from a real company.
The generating process gives propensity and persuadability different structure, which is what produces the gap. That is an assumption about the world rather than a finding: in a population where the effect is the same for everyone, the two rankings would perform identically. Whether your population looks more like the first or the second is an empirical question, and a holdout is how you answer it.