Calibrating an MMM to Experiments
Which experiment to run, how to carry its estimate into the model, and how hard to lean on it. Anchoring a channel's level drags its slope down with it, and the slope is what the media plan runs on.
A geo holdout takes six weeks and answers one question about one channel. A mix model answers every question about every channel and cannot be trusted on any of them. Calibration is the trade: spend the experiment where the model is weakest and let it discipline the rest.
Three decisions sit inside that sentence and almost nobody writes them down. Which experiment to run, how to carry its answer into the model, and how hard to lean on it once it is in there.
Which experiment to run
The standard advice is to hold a channel out. That is one design, and it answers one of the two questions a mix model gets asked.
An on-off holdout measures a level. Turn the channel off in matched markets, compare, and you learn what it contributed at the spend you were running. This is the easiest test to power, the easiest to explain, and the one almost every program runs.
A spend-change test measures a slope. Run the channel at two different levels across matched market groups, say 60% and 130% of plan, and the contrast between them estimates what an extra dollar buys near current spend.
The second is worth more if the model is going to set budgets, because allocation runs on slopes. It is also harder: a smaller contrast needs more markets or more weeks for the same detectable effect, and a finance team finds “we spent more in half the country” a stranger sentence than “we paused.”
Getting the estimate into the model
The experiment ran in 40 markets for 6 weeks at a weekly intensity nothing like the national plan. So a raw conversions-per-dollar rate does not transfer. Cost per conversion does better, and here it is $95, which implies a national contribution of 37,963 conversions against the 79,516 the unanchored model believes and 39,107 in truth.
Then comes the choice that decides everything downstream. The anchor can be applied in two ways.
Level only. Fix the channel’s contribution to the experimental figure and keep the adstock and saturation the unanchored model already fitted. The curve is rescaled. This is what a prior on a channel’s return does on its own, and it is the default in most calibrated models, Meridian included.
Curve refit. Fix the contribution and re-estimate the channel’s adstock and saturation subject to that constraint, so the shape is chosen under the anchor rather than inherited from the fit that produced the wrong number.
They report identical contributions. They do not produce the same media plan.
How hard to lean on it
A prior has a width, and the width decides how far the model moves toward the experiment. Below, that is expressed directly as the weight the experiment carries: zero is the unanchored fit, one is the hard anchor the worked example uses.
- True slope
- Anchor the level, refit the curve
- Anchor the level only
Show the data
| Weight on the experiment | True slope | Anchor the level, refit the curve | Anchor the level only |
|---|---|---|---|
| 0.0 | 3.7 | 3.4 | 3.4 |
| 0.1 | 3.7 | 3.2 | 3.2 |
| 0.2 | 3.7 | 3.1 | 3.1 |
| 0.3 | 3.7 | 2.9 | 2.9 |
| 0.4 | 3.7 | 2.7 | 2.7 |
| 0.5 | 3.7 | 4.0 | 2.5 |
| 0.6 | 3.7 | 3.7 | 2.3 |
| 0.7 | 3.7 | 3.8 | 2.2 |
| 0.8 | 3.7 | 3.5 | 2.0 |
| 0.9 | 3.7 | 3.1 | 1.8 |
| 1.0 | 3.7 | 3.8 | 1.6 |
This is the mechanism, and it is worth being slow about. The experiment says the channel contributed less than the model thought. Under a level-only anchor the whole curve is scaled down to match, so the height falls and the gradient falls with it. The model now believes the next $1,000 buys 1.63 conversions when it buys 3.67, and it stops funding a channel it should be funding.
Refitting the curve under the same constraint keeps the slope at 3.75. Same experiment, same contribution, a different answer about the next dollar.
- Best available at this bound
- Anchor the level, refit the curve
- Anchor the level only
Show the data
| Weight on the experiment | Best available at this bound | Anchor the level, refit the curve | Anchor the level only |
|---|---|---|---|
| 0.0 | 38.3 | 30.1 | 30.1 |
| 0.1 | 38.3 | 29.1 | 29.1 |
| 0.2 | 38.3 | 27.9 | 27.9 |
| 0.3 | 38.3 | 26.3 | 26.3 |
| 0.4 | 38.3 | 24.9 | 24.9 |
| 0.5 | 38.3 | 34.0 | 23.0 |
| 0.6 | 38.3 | 32.6 | 20.9 |
| 0.7 | 38.3 | 32.6 | 20.3 |
| 0.8 | 38.3 | 31.1 | 20.3 |
| 0.9 | 38.3 | 29.1 | 19.6 |
| 1.0 | 38.3 | 33.5 | 17.6 |
Both routes report the same improvement in contribution accuracy: the anchored channel goes from 103% wrong to -3% wrong as the weight goes to one, and that is the number a calibration gets sold on. The plans differ by 15.9 conversions a week at full weight, and nobody reports plan quality.
The two lines meet below a weight of about
0.5because until the anchor pulls the total far enough from the fitted one, the best-fitting shape under the constraint is the shape that was already there. Do not read the bump in the middle of the refit line as a sweet spot. The fitting routine is coordinate descent over a grid, so it lands in different local optima at different weights, and the wiggle is that rather than a finding. Read the ends.
What has to be set up before any of this
Calibration cannot rescue a model that was never in a position to be identified.
- Stable channel definitions across the whole window. A channel that was renamed, merged, or re-scoped midway is two channels wearing one name, and no prior fixes that.
- The non-media drivers, explicitly. Price, promotion, distribution, seasonality, and a holiday term. Everything you omit gets absorbed by whichever channel moves like it, which is usually the one that spikes in Q4.
- Spend variation that is not just the calendar. Flighting, staggered launches, regional differences, an actual pause. This is the raw material, and a smoothly planned three years contains almost none of it.
- Geo-week rather than national-week granularity where you can get it. More variation, and it lets a geo experiment be read in the same frame as the model instead of being transported into it.
- One outcome, and the one the business optimizes. Modeling net growth when the decision is about gross adds hands the model a churn problem it was not built to separate.
- Wide priors on everything you have not measured. A prior is a claim. The default should be a weak one, tightened only where an experiment earned it.
- Two validations, kept apart. Out-of-time fit catches overfitting. Experiments catch confounding. Passing the first says nothing about the second, and reporting the first as though it did is the most common way a mix model gets oversold.
A cadence rather than a project
One calibration is a one-off. What makes the model trustworthy is a standing rotation.
Run one experiment a quarter. Choose the channel by contribution multiplied by how little you know about it, rather than by whoever is arguing loudest, which usually means the largest channel first and then the one whose modeled contribution moves most when you change the specification. Refit when a new estimate lands, and keep the anchors visible: a model carrying four old priors nobody remembers setting is a model nobody can debug.
Context
Best suited for
Programs already running a mix model that has to set budgets, where at least one geo experiment per quarter is affordable and the largest channels are contested.
Data requirements
- An experimental estimate expressed as cost per incremental conversion, not a per-dollar rate, so it survives the change of scale
- The test window, geography, and intensity recorded alongside the estimate, since the prior is only as transportable as those
- A model whose curve parameters can be refit under a constraint rather than only rescaled
- A held-out time period kept separate from any of this, for the overfitting question
What changes by context
- B2C ecommerce
- Promotional calendars make the transport assumption hardest here. An experiment run in a promo-heavy window estimates a channel under conditions the rest of the year does not share, so the prior deserves to be wider than the standard error suggests.
- B2C subscription
- The outcome arrives on a lag longer than most geo tests run, so the experimental estimate is usually a partial read. Anchoring on a truncated outcome pulls the whole model down, and the fix is a longer test rather than a wider prior.
- Marketing-led
- The cadence is affordable and the channels are large enough to test, so the rotation is realistic. This is the setting where a standing calibration program compounds rather than being one heroic quarter.
When it breaks down
- Channels that cannot be geo-split, such as national sponsorships or anything bought at a single national placement.
- Experiments too underpowered to distinguish the estimate from the model, where the prior is wide enough to change nothing.
- Models whose implementation exposes only a return prior, with no way to refit the curve under it. Then the level-only column above is the one you get.
- Programs where the model is refit continuously and nobody records which anchors are live.
What this does not establish
- That refitting the curve is always better. It is better here at every weight above the point where the two diverge, on both contribution error and plan value. On a program where the fitted shape was already close, there is nothing to recover.
- That any particular prior weight is correct. The interior bump on the refit line is a fitting artifact rather than an optimum, and the defensible reading is the comparison at the ends.
- That the size of the loss transfers. Giving back 11 conversions a week is a property of these curves and this anchor. The direction is the finding, not the magnitude.
- That calibration makes a mix model causal. It disciplines one channel with one experiment. Every other channel is still identified by the same history it always was.
- That an experiment estimated on one window applies to another. Nothing here tests transport, which is the assumption the whole method rests on.
What to test next
If a calibrated model is already running, check which of the two routes it took. Ask whether the anchored channel’s adstock and saturation changed when the prior was applied. If they did not, the model is on the level-only line, and the scenario planner in what a mix model can and cannot do shows what that costs.
The experiment worth queuing is a two-level spend test on the largest channel, because the quantity you are about to optimize is a slope and no on-off holdout estimates one. Design it the way Designing a Credible Geo Holdout designs the on-off version, with the second cell set high enough that the contrast clears your detectable effect rather than at whatever budget was left over.
How the prior weight maps to a prior width Code
Under a normal prior and a normal likelihood, the posterior mean is a precision-weighted average of the two, and the weight on the prior is its precision over the total. The sweep above uses that weight directly, from 0 to 1, because it is the quantity that has a consequence. A tool that asks for a prior standard deviation is asking the same question in units that depend on the parameterization, which is why two teams can set what they believe is the same prior and get different answers.
Practically: a weight near zero means the experiment is decoration, and a weight near one means a 6-week reading from 285,000 dollars of spend is overriding three years of national history. Neither end is obviously right, and the argument for sitting away from one is the transport assumption rather than anything in the fit.
Where these figures came from Code
The same simulated advertiser, fitting code, and geo holdout as the experiment-calibrated mix model, so the numbers here describe the same company. The generator imports that one rather than re-simulating.
Plans are scored by the same allocation rule the scenario planner uses, equalizing marginal return under a 30% bound on how far any channel may move, then evaluated against the curves that generated the data. That last step is the one no real calibration can perform, and it is the only reason the cost of a level-only anchor is visible here at all.