An Experiment-Calibrated Marketing Mix Model

Five channels, roughly $6197k a year, and a planning cycle that has to split next quarter’s budget between them. Three years of clean weekly history are available.

The request is the usual one: build a mix model, tell us what each channel contributed, and recommend an allocation.

Model fit. It is the number every mix model is sold on, and this one is excellent: R² of 0.964 against three years of weekly conversions, with mean absolute percentage error of 1.3%.

A chart like this ends most model reviews. It should not end this one.

Fit measures whether the model can reproduce the outcome. It says nothing about whether it has divided the credit correctly, and those are separable problems.

Media spend does not vary independently of everything else. Budgets rise in Q4 and so does demand. Worse, branded search spend is bid against demand: when more people search for the brand, the channel spends more, automatically, every week. Its spend and the underlying demand curve are nearly the same shape.

Given two variables that move together, least squares has no way to tell which one is doing the work. It will assign the credit in whatever combination minimizes error, and a great many combinations minimize error almost equally well.

Available. 156 weeks of spend by channel, conversions, a price index, promotional periods, and the calendar.

Missing. Any variation in spend that was not decided in response to something else. Every budget in this history was set by people reacting to demand, competitors, and last quarter’s results. That is the whole problem: observational media data contains almost no exogenous variation, and exogenous variation is what identifies a causal effect.

Also available, and this is what makes the second half of the analysis possible: a geo holdout on paid social measuring incremental cost per conversion at $95, with a 95% interval from $66 to $173.

For each channel, the incremental conversions caused by its spend, and the shape of the curve relating the two, so that a marginal return can be computed at any budget level.

Not the share of conversions that correlate with each channel. That quantity is always estimable and is not what anyone is asking for.

Fit the model twice.

Model one: history alone. Geometric adstock and a concave saturation curve per channel, with decay and half-saturation found by grid search and the coefficients solved by least squares at each candidate. Controls for price, promotions, trend, annual seasonality, and a holiday term. Media coefficients constrained to be non-negative.

Model two: anchored to the experiment. Identical, except that paid social’s contribution is fixed so its implied cost per conversion matches the geo holdout. The remaining channels are fitted against what is left.

Because this is simulated, both can be scored against a truth no real project has.

What the history-only model concluded

Branded search produced 29,369 conversions. The model credits it with 74,465, an overstatement of 154%. That is the channel whose spend tracks demand, and it has absorbed a large share of the baseline.

That is one draw of the noise. Step nine refits the model a hundred times on resampled residuals and asks how stable that 154% is; the answer is that it is not, and the direction it fails in is the finding.

The error does not stop there. Because branded search takes credit that belongs elsewhere, every other channel is squeezed. Affiliate is understated by 65%, which matters more than it sounds: a model that cannot see a channel will recommend switching it off.

What the anchor changes

Fixing paid social to the experimental estimate brings that channel to within 3% of the truth, from 103%. Mean absolute error across all five channels falls from 76% to 61%.

That is an improvement and it is not a repair. Branded search is still overstated, slightly more than before, because the credit paid social gave up had to go somewhere and branded search is still the channel that most resembles the unexplained variation.

That interval is the identification problem made visible. Many divisions of credit between branded search and the demand it is bid against reproduce the series equally well, so the fit statistics are the same across the whole range and the noise decides where the estimator lands. The +154% above is where it landed on this draw. The median across refits is −52%, so the model does not even overstate the channel reliably: it is unidentified, and an unidentified estimate has no reliable direction. The other four channels swing about as widely.

A real account would produce the same bootstrap interval and no truth column to read it against. Which is the point: an interval on a mix model coefficient describes sampling noise, and sampling noise was never the problem.

The two models disagree sharply about the past and much less about the future.

Both models recommend the same broad moves: cut branded search by roughly a third, cut affiliate hard, and roughly double online video. Across the whole budget they differ by about 7%.

That is worth being clear about rather than overselling. Calibration substantially changed the story about what happened, and changed the recommendation about what to do next only modestly, because the allocation rule depends on the shape of each curve near current spend and both models recovered similar shapes.

Act on the direction, not the decimals. Fund the next experiment on branded search.

• Both models agree the current allocation is wrong in the same way. Four of the five moves point the same direction under both, including the three largest. Make those. Paid social is the exception, and it is the channel the anchor moved, so it is the one to hold: see what a mix model can and cannot do for why anchoring a level can leave the slope worse than it found it.
• Do not report channel contributions from the history-only model. They are wrong by a factor of two on the largest line, and reporting them will eventually be discovered.
• The affiliate recommendation is the one to hold back on. Both models want it cut to near zero, and both underestimate it by 65%. A model that cannot see a channel recommends deleting it, and that recommendation should not be executed on model evidence alone.
• Branded search is the channel worth testing next, because it carries the largest spend and the largest identification problem, and because a holdout there would anchor the term currently absorbing everyone else’s credit.

1. A geo holdout on branded search. Largest spend, worst identification, and the term currently absorbing the baseline. One test would improve the model more than another year of history.
2. A spend-level test rather than an on-off test. Two arms at different budgets estimate the slope, which is what the allocation rule needs. An on-off test gives one point.
3. A standing calibration cadence. One experiment per quarter, rotating across channels, keeps several anchors current instead of one going stale.
4. Hold out a year. Refit on the first two years and check the third. It will not detect the identification problem, but it will catch overfitting, which is a different failure worth ruling out.