The Budget Decision Log

Last quarter the plan moved a fifth of the video budget into search on the model’s recommendation. This quarter the model, refitted on a quarter more of data, recommends moving some of it back. Nobody in the room can say whether the first move worked, because nobody wrote down what it was supposed to do.

The model has now made two recommendations and been checked against zero outcomes. It will make a third next quarter.

The decision it encodes

What did we expect this budget move to do, and did it?

Every other component in this system produces evidence for a decision. This one records the decision, with its evidence attached, so the next readout has something to score. The lifecycle system’s decision log is what makes its policy evaluable. Paid media has no equivalent by default: budgets move, the model refits, and the recommendation and the outcome never meet.

A recommendation nobody scored is a forecast nobody made

A mix model’s reallocation is a prediction. It says that moving this money will produce roughly that many more conversions, with a range. A prediction that is recorded before the outcome, and compared to the outcome afterwards, is the only thing that ever tells you whether the model deserves the budget it is directing. A prediction that is not recorded is indistinguishable from one that was never made, and the model gets credit for whatever happens next.

That is a forecasting discipline older than marketing measurement: state the forecast, state its range, wait, compare. It is how weather models and demand planners are held to account, and it is almost never applied to the model that decides where the media budget goes.

What one entry in the log carries

Field	What it decides
The move	Each channel’s spend before and after, and the date it took effect. What was decided, rather than what was proposed.
The curves behind it	The model version, its fitted parameters, and which experimental anchors it carried. So the recommendation can be reproduced, and so a bad move can be traced to a bad anchor rather than argued about.
Constraints binding	Which contracts, minimums, or rate limits held the plan away from the unconstrained optimum, and their shadow prices. What it would have been worth to renegotiate.
The prediction	Incremental outcome the move was expected to produce over a stated horizon, with a predictive interval that includes outcome noise, not only the model’s uncertainty about the effect. Written before the move ships.
Staging	Whether the move ran in a subset of markets first. A staged move is a geo test; an unstaged one is a bet.
The outcome	What the horizon delivered, from the same outcome definition the prediction used, with its own interval where the move was staged.
The verdict and score	Consistent with the forecast, in tension with it, or unreadable at this precision, plus an interval score, and what the next calendar slot should ask because of it.

Staging is what makes a move readable

A reallocation that lands everywhere at once can be checked against its forecast, and that is worth doing. It cannot be read as a causal estimate, because everything else changed in the same quarter too. A reallocation that runs in a subset of matched markets first, with the rest held at the old level for a rotation, is a geo test whose treatment is the budget move itself.

That is the edge on the diagram. A staged move puts a channel at a spend level the history never covered, in markets where the comparison exists, which is the variation the response curves could not identify from observational data and the calendar would otherwise have to buy with a dedicated test. Recording it here is what turns that variation into evidence the next refit can use.

What the log feeds

Three places, and none of them can be fed without it.

The calendar. A move whose outcome sat in tension with its forecast is a channel the model was wrong about, and that error is the uncertainty term in the calendar’s ranking, weighted by the spend on the channel and by whether the budget decision would change. The largest miss is not automatically the next test. Ordering the queue needs an estimate of how wrong the model is, and the log is the only source of one that does not come from the model itself.

Staged decisions update the evidence base. Unstaged misses update the test queue. That rule is what keeps a forecast miss on a move made in an ordinary quarter from being written into the curves as if it were a measured effect.

Calibration. A staged move is a geo test, so its readout is an anchor with a date, and it lands in calibration the same way a dedicated test does.

The next reallocation. A model with a record of forecasts and outcomes can be given a calibration score, and a planner can be told how much to trust the decimals. A model without one is trusted by default, which is how the third recommendation gets made on the strength of two that were never checked.

Maturity, honestly staged

Stage 1: moves recorded with a date and a reason. A change log, which most programs do not have, and enough to answer what changed when.

Stage 2: a prediction with an interval, written before the move. The model’s recommendation carried its own expected effect; writing it down costs nothing and changes what the next quarter can say.

Stage 3: outcomes scored at the stated horizon, automatically. The readout runs on a schedule, from the stored definition, and the verdict lands in the same log entry.

Stage 4: staged moves by default. Reallocations above a stated size run in a subset of markets first, and the log entry is also a calendar entry and a calibration anchor.

Scoring a model's forecasts Technical

Coverage is the first check and the weakest one. Across many entries, the share of outcomes inside the model’s 80% intervals should be near 80%, and a model landing half the time is overconfident. But coverage rewards width: an interval from minus infinity to plus infinity covers everything and decides nothing. A proper scoring rule grades calibration and sharpness together. The interval score is the simplest: the interval’s width plus a penalty proportional to the distance by which the outcome fell outside it. Log scores and CRPS do the same for a full predictive distribution. Track the score alongside bias, coverage, and mean width, by channel, by size of move, by horizon, and by staged against unstaged, and compare it to a benchmark such as a forecast of no change. A model that cannot beat that benchmark has not earned the budget it directs.

Two intervals are easy to confuse here. A credible interval describes uncertainty about the effect. A predictive interval describes the range of outcomes that might actually be observed, which adds the outcome’s own noise, and it is the one a realized figure should be scored against. Scoring an unstaged move against a credible interval will make an accurately estimated effect look wrong every quarter, because ordinary variation in the outcome was left out. For a staged move the realized figure has an interval too, from the randomization, so the verdict weighs both: consistent with the forecast, in meaningful tension with it, or unreadable at the precision the test had. That is calibration in the forecasting sense, and it is a different property from the experimental calibration the estimation stage performs.

Unstaged moves give a weak version of this, because the realized outcome includes everything else that happened. The comparison is still worth making, since a model that is repeatedly wrong in one direction on unstaged moves is telling you something, but a miss on an unstaged move is a prompt for a test rather than a finding.

The log also makes the model’s own history reproducible. A model version, its parameters, and its anchors are enough to regenerate the recommendation later, which is what lets a bad quarter be traced to the anchor that caused it rather than to the model in general.

Context

Best suited for

Programs where a model recommends budget moves on a cadence and the moves are made, and where nobody can currently produce the model's record of predicted against realized effects.

Data requirements

• The reallocation output, with the model version and anchors that produced it
• Spend by channel, market, and day after the move, from the spend spine, so staging can be verified
• The outcome at the stated horizon, computed from the definition stored with the prediction
• A place the log lives that the calendar and the calibration step both read

What changes by context

B2C ecommerce

Horizons are short enough that most moves can be scored within the quarter they were made, which is what makes a quarterly cadence a loop rather than a lag.

B2C subscription

The outcome that matters lands months after the move, so each entry carries two predictions: one on signups the planner can score soon, one on retained value it cannot, and the second is the one that decides whether the move was right.

Marketing-led

Where the log has the most to say, since channel mix is the main lever and the model is trusted with the most money. It is also where an unscored model is most expensive.

When it breaks down

• Programs that move budget rarely and by small amounts, where the log fills slowly and nothing in it is readable for years.
• Organizations where the model is advisory and the budget is set elsewhere, so the moves recorded are not the model’s and cannot score it.
• Outcomes that cannot be measured at any horizon inside a planning cycle, where the log records predictions nobody can check.