Why Agentic Marketing Still Needs Holdouts
Two agents, the same budget and the same population, differing only in what they are scored on. One improves its dashboard by 65% while destroying value, and the one doing the right thing looks worse.
The case for handing a marketing program to an autonomous system is a good one, and it is worth stating properly before arguing with it.
An agent runs the loop continuously. It tries things, sees what happened, and adjusts, at a cadence no team can match. It does not get bored, does not have favourites, and does not defend last quarter’s plan. Against that, a holdout looks like a static-campaign idea: why withhold treatment from anyone when the system is already learning from live outcomes every day?
Because learning from live outcomes tells you which direction it is improving in only if the outcome you handed it is the right one.
Two agents, one difference
Below are two runs of the same agent on the same population, with the same budget of 40,000 contacts a cycle, the same allocation rule, and the same exploration. The only difference is the number each one is trying to increase.
The first is scored on attributed conversions: how many people it contacted went on to convert. That is the number an analytics platform produces daily and the one most readily available to point an agent at.
The second is scored on incremental conversions. It withholds 20.0% of the people it would have contacted in every segment and scores itself on the difference. That estimate is not free: it makes fewer contacts and forgoes the effect on the people it withheld.
What the dashboards showed
- Agent scored on attributed conversions
- Agent scored on incremental conversions
Show the data
| Cycle | Agent scored on attributed conversions | Agent scored on incremental conversions |
|---|---|---|
| 1 | 154.1 | 154.0 |
| 2 | 160.2 | 151.5 |
| 3 | 167.0 | 149.3 |
| 4 | 173.4 | 146.8 |
| 5 | 179.9 | 144.7 |
| 6 | 186.3 | 143.1 |
| 7 | 192.1 | 141.1 |
| 8 | 197.7 | 139.1 |
| 9 | 203.1 | 138.1 |
| 10 | 208.5 | 136.5 |
| 11 | 213.2 | 135.4 |
| 12 | 217.7 | 134.5 |
| 13 | 222.0 | 133.1 |
| 14 | 225.7 | 132.1 |
| 15 | 229.7 | 131.6 |
| 16 | 233.0 | 130.6 |
| 17 | 235.9 | 130.3 |
| 18 | 239.0 | 130.2 |
| 19 | 241.0 | 129.6 |
| 20 | 244.0 | 128.9 |
| 21 | 245.5 | 128.7 |
| 22 | 247.3 | 128.5 |
| 23 | 249.8 | 128.5 |
| 24 | 251.1 | 128.8 |
| 25 | 252.5 | 128.2 |
| 26 | 254.2 | 128.1 |
On this chart the attributed agent is the success story. It found the segments that convert and moved budget into them, and its reported efficiency climbed from 154 to 254 conversions per thousand contacts. Any weekly review would show it working.
What actually happened
- Agent scored on attributed conversions
- Agent scored on incremental conversions
Show the data
| Cycle | Agent scored on attributed conversions | Agent scored on incremental conversions |
|---|---|---|
| 1 | 27.5 | 27.5 |
| 2 | 26.7 | 29.4 |
| 3 | 25.7 | 31.4 |
| 4 | 24.7 | 33.3 |
| 5 | 23.6 | 35.3 |
| 6 | 22.4 | 37.1 |
| 7 | 21.2 | 38.9 |
| 8 | 20.0 | 40.6 |
| 9 | 18.7 | 42.3 |
| 10 | 17.5 | 43.8 |
| 11 | 16.2 | 45.3 |
| 12 | 15.0 | 46.6 |
| 13 | 13.8 | 47.9 |
| 14 | 12.7 | 49.1 |
| 15 | 11.6 | 50.2 |
| 16 | 10.6 | 51.2 |
| 17 | 9.6 | 52.2 |
| 18 | 8.7 | 53.0 |
| 19 | 7.9 | 53.8 |
| 20 | 7.1 | 54.5 |
| 21 | 6.3 | 55.2 |
| 22 | 5.6 | 55.9 |
| 23 | 5.0 | 56.4 |
| 24 | 4.4 | 56.9 |
| 25 | 3.8 | 57.4 |
| 26 | 3.3 | 57.9 |
Over 26 cycles the attributed agent produced $737,001 of net value and the incremental agent produced $2,857,106, which is 3.9 times as much from the same budget, after paying for its own holdout.
Where the budget went
| Segment | Converts anyway | True uplift | Attributed agent | Incremental agent |
|---|---|---|---|---|
| Recent buyers | 30% | −1.0pp | 58.8% | 2.4% |
| Highly engaged | 24% | +1.0pp | 26.6% | 3.1% |
| Regular | 12% | +4.0pp | 5.7% | 9.9% |
| Lapsing | 6% | +7.0pp | 4.0% | 63.8% |
| Dormant | 3% | +5.0pp | 2.7% | 18.1% |
| Long dormant | 1% | +0.5pp | 2.2% | 2.7% |
The attributed agent put 85.4% of its budget into the two least persuadable segments, one of which has a negative effect. It was not malfunctioning. Those segments produce the most conversions among people you contact, and that is what it was told to maximize.
The mechanism, stated plainly
An attributed reward is a measurement taken entirely among treated people. It can tell an agent that action A beat action B, provided both were tried. It can never tell it that acting beat not acting, because not acting was never an arm.
So the agent optimizes the only comparison available to it: between people it contacted. Contacting people who were already going to convert wins that comparison every time. The agent is not being fooled. It is succeeding at the task it was given, and the task was wrong.
The uncomfortable part
Look at the two charts together. The agent doing the right thing has the worse dashboard, and it has it for the whole run.
Its reported conversions per contact fall −17% while its real effect more than doubles, because moving budget toward persuadable people means contacting people who convert less often in absolute terms. An organization comparing the two on the metric it already has would switch off the one creating value, and would have a chart to justify it.
That is the practical case for the holdout, and it is not about statistics. Without a control there is no number in the building that distinguishes these two agents, and the number everyone does have points the wrong way.
The minimum that fixes it
Two things, both of which the systems section specifies in detail.
A persistent holdout enforced at assignment, which the agent cannot observe or select into. If the agent is the thing that skips the holdout, the holdout stops being random the first time the agent is retrained. Governance for agentic campaigns covers why that enforcement point matters and where it sits.
An incremental objective rather than an attributed one, which means something has to be producing incremental estimates on a schedule. Automated measurement is that thing, and it is the reason the autonomy is the last component in the chain rather than the first.
Context
Best suited for
Programs about to hand allocation, targeting, or timing to an autonomous system, especially where the obvious objective is a number the analytics platform already produces.
Data requirements
- A holdout enforced upstream of the agent, invisible to it and stable across retraining
- An incremental estimate per segment or action, refreshed at least as often as the agent adapts
- Decision logs carrying what was chosen, what else was available, and the probability each carried
- Guardrail metrics on a cadence fast enough to trigger a rollback rather than a post mortem
What changes by context
- Lifecycle-led
- Decision volume is highest and the anti-correlation between engagement and persuadability is sharpest, because the most engaged subscribers are also the ones least in need of persuading. This is the setting where the gap opens fastest.
- Product-led
- The agent acts inside the product, so its actions change the surface the outcome is measured on. Withholding means withholding a product experience, which is a harder conversation internally and the same requirement.
- B2C subscription
- A retention agent rewarded on retention will find discounting quickly, because it works and the margin cost lands in a different report. A hard ceiling on discount depth matters more here than any other guardrail.
When it breaks down
- Populations where effects are roughly uniform, since both objectives lead to the same allocation and the holdout only confirms it.
- Agents acting on a population small enough that a holdout leaves neither arm estimable.
- Programs with heavy spillover between people, where a person-level holdout is contaminated and geo assignment is the only design left.
- Decisions that are obligations rather than choices, such as transactional messaging, which do not belong on this comparison at all.
What this does not establish
- That attributed rewards always destroy value. The gap here follows from baseline and uplift pulling in opposite directions. Where they agree, both agents do the same thing.
- That the size of the gap transfers. Nearly four times the net value is a property of these six segments and this budget, not a rate to expect.
- That this is how a real agent learns. The allocation rule here is exponential weights over six fixed segments, far simpler than anything in production. The point is the direction the reward pulls in, and that direction does not depend on the learning rule.
- That a 20% holdout is the right size. It was chosen so the estimate is precise enough to steer on, and sizing it properly is a power calculation against your own effect and base rate.
- That an incremental objective makes an agent safe. It fixes what the agent is aiming at. Guardrails, bounded actions, and consent enforcement are separate problems.
What to test next
If an agent is already live and was never held out from anyone, start withholding now. It does not require retraining or a new objective. A random slice excluded at assignment, from today, produces a defensible estimate of what the agent is worth within a cycle or two, and it is the only number that will still be meaningful after the next three versions ship.
If an agent is being planned, the order matters more than anything in this lesson. Get the incremental estimate running first, on the segments the agent will act over, and point the agent at that. Installing the autonomy before the measurement produces an extremely efficient machine for optimizing the wrong quantity.
How the two agents work Code
Six segments with a fixed population, a baseline conversion rate, and a true effect per contact. Both agents allocate 40,000 contacts a cycle using exponential weights on whatever signal they can see, with a floor on each segment’s share so exploration never stops entirely. Outcomes are drawn binomially, and every figure is the mean of 120 replications so the lines show the direction of learning rather than one run’s noise.
The attributed agent sees conversions per contact among the people it contacted. The incremental agent withholds 20.0% of each segment’s allocation and sees the difference between contacted and withheld. Its holdout is paid for out of its own budget, so the comparison charges it for the measurement rather than granting it for free.
Net value is the true incremental conversions times $85 of margin, less $1 per contact. These are not measurements from a real company, and the population was constructed so that baseline propensity and persuadability disagree, which is the assumption stated above.