Calibration

The geo test came back. Paid social is incrementally worth about 0.6 times what the mix model has been claiming, with an interval that comfortably excludes the model’s number.

The deck reports both. The model keeps its coefficient because refitting is a quarterly job and the test arrived in week six. The plan is built on the model. Six months later the same test is run again, disagrees again, and is reported again, and nothing in the system has a mechanism for the disagreement to change anything.

The decision it encodes

When the experiment and the model disagree, which one moves?

Somebody has to answer that in advance, in writing, because answering it after seeing which direction the disagreement points is how a measurement program becomes a negotiation.

First check they are answering the same question

Most disagreements are definitional before they are evidential, and that is the cheapest thing to rule out.

An experiment measures a specific change, in specific markets, over a specific window. A model coefficient describes an average effect across all markets over the fitted period. Those differ legitimately when the test ran in markets that are not typical, when the outcome window was shorter than the effect, when the test moved spend from a level the average does not represent, or when one counts revenue and the other counts orders.

Before an experiment becomes a prior
Check	Why it decides whether the estimate transports
Spend range	Was the tested spend change in the range the model is being asked to decide over? A blackout speaks to the whole channel; a marginal scale-up speaks to the next dollar.
Recency	Has the creative, the auction, the audience, or the bidding changed since? A test on a channel that has been restructured is a test of a channel that no longer exists.
Markets	Were the tested markets representative of where the spend goes, or the ones operations could spare?
Outcome and window	Same outcome definition and a comparable horizon, or the gap is definitional before it is evidential.
Design	Blackout, scale-up, or marginal test, since each identifies a different point on the curve.
Adjustments	What was done about duration, recency, geography, and spend scale before the estimate was placed as a prior, stated rather than implied.
Also used for validation?	If this test calibrated the model, it cannot also be quoted as independent evidence the model is right.

A test that fails a row can still be used. What it cannot be is placed as a prior on a quantity it did not measure without the adjustment written down.

Calibration is a prior, weighted by precision

Once alignment is established, the mechanism is straightforward. The experiment’s estimate becomes a prior on the model’s coefficient for that channel, and the tightness of that prior follows the experiment’s own precision.

A test with a narrow interval pulls the coefficient most of the way. A test with an interval spanning zero to double should barely move it, and saying so protects the model from being dragged by an underpowered result that happened to land far from it. The strength of the prior is the setting where teams either express what they know or quietly overwrite one estimate with another. Calibrating a mix model to experiments works through what different prior strengths do to the same fit.

A calibrated model is not a validated model

Worth stating plainly, because the language invites the confusion. Calibration anchors the channels you tested. It does nothing for the channels you did not, and it does not make the model’s functional form correct or its extrapolations safe.

What it does provide is a defensible claim about the tested channels and a visible gap around the rest, which is more useful than a uniform confidence that was never warranted anywhere. The channels that remain uncalibrated should be the ones re-entering the test calendar queue, which is how the loop closes rather than stalling here.

Validation needs evidence the model has not already absorbed. A test that was placed as a prior is part of the model, and quoting it afterwards as proof the model is right is circular. Either hold some tests back from calibration to score the model against, or validate prequentially: the model forecasts first, in the decision log, and is scored on evidence that arrived later.

Say what changed and when

A model that has been calibrated is a different model from the one before it, and plans built on the two are not comparable. The record should carry which channel was calibrated, against which test, with what prior strength, on what date, so that a reallocation from last quarter can be understood in terms of the evidence available then rather than looking like a reversal.

Maturity, honestly staged

Stage 1: experiments and model coefficients reported side by side, with their estimands. No mechanism yet, and definitional disagreements stop being mistaken for evidential ones.

Stage 2: a written rule for what happens on disagreement. Decided before the result exists, which is the whole point.

Stage 3: experiments entering as priors, weighted by their precision. The model absorbs the evidence rather than sitting alongside it.

Stage 4: calibration status tracked per channel, feeding the queue. Uncalibrated and stale channels are visible, and they are what the next test gets scheduled against.

How the experiment enters the fit Technical

The clean version is to place a prior directly on the quantity the experiment measured, rather than on a raw coefficient. A geo test that cut a channel and observed a change in outcome implies an incremental return over the tested spend range; the model can be asked to reproduce that quantity, with a prior centered on the test estimate and a scale taken from its standard error.

Two details decide whether this behaves. The prior belongs on a quantity the experiment identified, which is usually an effect over a spend range rather than a single coefficient in a transformed space, since λ and h are not separately measured by a test that turned a channel off. And the scale should reflect the experiment’s own uncertainty rather than being tightened to make the fit obey, which is the common failure and produces a model that agrees with the test because it was told to.

When several tests exist for one channel across time, they are more useful as a series than as a replacement for each other. Two tests a year apart that agree tell you the effect is stable, which is a claim no single test can make, and disagreement between them is information about decay rather than about which one to trust.

Context

Best suited for

Programs running both a mix model and experiments, where the two currently produce separate numbers that get reconciled in whichever meeting notices, and where planning uses one of them without stating which.

Data requirements

Experiment estimates with intervals, windows, markets, and spend levels attached
A model whose priors can be set per channel rather than only globally
Estimands written for both, so definitional gaps are separable from evidential ones
A dated record of each calibration, since a recalibrated model is not the one that produced the plan before it

What changes by context

B2C ecommerce: Tests are fast and cheap enough that several channels can be calibrated a year, so the binding constraint is keeping the estimands aligned across tests run under different promotional conditions rather than getting tests at all.
B2C subscription: The test reads on signups long before retained value, which is the outcome the model uses, so calibrating directly across them imports the retention assumption silently. Either wait for the longer outcome or make the conversion explicit and priced.
Marketplace: A test on one side changes conditions on the other, so its estimate includes an equilibrium effect the model may be attributing elsewhere. Calibrating the coefficient for one side to it can move the error into the other side rather than removing it.

When it breaks down

No experiments, where there is nothing to calibrate against and the model should be described as uncalibrated rather than dressed in priors.
Experiments too imprecise to move anything, where the action is to record the test and leave the coefficient alone rather than pretending the prior did work.
Channels restructured between the test and the fit, where the thing measured no longer exists in the model.
Organizations that will override the calibrated number when it is inconvenient, which makes the whole step decorative.