One Quarter of the Budget Decision Log, Scored

Last quarter the plan moved money four times on the model’s advice. Each move went into the decision log with the model version behind it, the incremental conversions it predicted over the quarter with an 80% predictive interval, and whether it ran in a subset of markets first. This quarter the log scores them. Nothing here was computed after the fact except the last two columns.

Every number is simulated from four invented moves. The shape is the claim.

The moves, scored

Q3 moves, read one quarter later · predictions carry 80% predictive intervals

Channel	Move	Predicted	Realized	Read as	Verdict
Online video	+$230k	+1,900+1,349 to +2,451	+1,483+1,142 to +1,824	Geo test, 20 markets staged	Consistent
Branded search	−$145k	−400−677 to −123	−114−114 to −114	Forecast against actual, unstaged	Tension, high
Affiliate	−$30k	−900−1,317 to −483	−1,591−1,591 to −1,591	Forecast against actual, unstaged	Tension, low
Paid social	+$25k	+210+88 to +332	+183+141 to +225	Geo test, 10 markets staged	Consistent

The score, and what two tensions in four say

The model’s quarter, scored

Coverage

2 of 4 nominal 80%

Mean interval width

684 conversions

Bias

−212 realized minus predicted

Interval score

1,390 vs 6,100 for no effect

The interval score is the interval’s width plus a penalty of ten times the distance by which the outcome fell outside it, so it punishes both a wide interval and a confident miss, which coverage alone cannot. Lower is better. The benchmark is a forecast of no effect carrying the same widths. The model beat it by a wide margin this quarter, and a model that stops beating it has stopped earning the budget it directs.

Coverage on its own would say the model’s 80% intervals contained the outcome 2 times in 4. Four moves is too few to call that overconfidence, and coverage is the wrong thing to optimize anyway, because an interval from minus infinity to plus infinity covers everything and decides nothing. The score is what accumulates: a model whose intervals keep landing half the time and whose score drifts toward the benchmark is a model whose reported uncertainty should be widened until the score recovers, and the planner is told that in the next cycle rather than trusting the decimals.

The direction of the tensions is the useful part. Branded search was cut by $145k on a prediction of −400 conversions and cost −114, which sits 1.3 predictive standard errors above the forecast: the model believed the channel more than the channel deserved, which is the pattern the calibrated mix model predicts for a channel bid against demand. Affiliate was cut by $30k on a prediction of −900 and cost −1,591: the model believed the channel less than it deserved, which is what a model does to a channel it cannot see. Both tensions are on unstaged moves, so neither is a measurement, and both are the reason the calendar exists. Staged decisions update the evidence base. Unstaged misses update the test queue.

The standing holdout, rotation two

Persistent geo holdout · Branded search · 10% of markets dark, rotating

Rotation 1

+4.7% −2.4% to +11.8%

Rotation 2

+6.1% −1.2% to +13.4%

Pooled

+5.4% +0.3% to +10.4%

Lift in total sales from the channel running, dark markets against lit, each rotation read after its washout. Neither rotation alone clears zero. Pooled they do, on the assumption that the channel’s effect held still across the half year, and the two rotations agree within their intervals, so the assumption survives this read. This is the live anchor calibration reads between dedicated tests.

The next slot

The calendar orders its queue by expected decision value, and the log supplies one of its inputs: for each move, the spend resting on the estimate multiplied by how far the outcome fell from the prediction, which is the uncertainty term in that ranking. The largest miss is not automatically the next test. A large miss on a small, capped channel changes no decision, and a modest one on the largest channel might. Here the two agree: Branded search carries the most spend among the moves in tension, its budget would move at either end of the tension, and it has never been tested directly. The standing holdout keeps its anchor current; the dedicated test is what would move its coefficient.

What this does not establish

• That the model is overconfident. Two tensions in four is consistent with an 80% interval, the interval score beat the benchmark comfortably, and the log needs a few more quarters before either verdict means anything.
• That cutting branded search cost less than predicted because the channel is weak. The move was unstaged, so the realized figure includes the quarter’s seasonality, a competitor’s launch, and everything else; the direction is a hypothesis for the next test, and the test is scheduled.
• That affiliate should be restored. The miss says the model under-values it, which the calibrated model example already found; how much it is worth is a question for a staged move, not a reversal.
• Anything about a real media plan. Four moves were invented so the quarter would show two staged moves consistent with their forecasts and an unstaged tension in each direction.

How this readout was produced Code

Four moves with stated spend changes, predicted incremental conversions, the model’s standard error on each effect, and planted true effects. Staged moves are realized as the true effect plus geo-test noise, with a 95% interval from it. Unstaged moves are realized as the true effect plus the noise of a national forecast against actual, which is several times larger; the model’s 80% interval for those carries that noise too, which is what makes it a predictive interval rather than a credible interval on an effect. The verdict is graded on the gap between realized and predicted against the combined uncertainty of both: consistent inside 1.28 standard errors, tension outside. The interval score is Winkler’s, at the 80% level, and the benchmark applies it to a no-effect forecast with the same widths.

The holdout’s rotations are two stated estimates with standard errors, pooled by inverse variance; the agreement check asks whether they differ by more than the interval on their difference. The next slot ranks moves by spend multiplied by proportional miss, which is the uncertainty-times-money term of the calendar’s ordering; the check that the decision would change is made in prose here and belongs in the calendar’s own scoring.

Regenerated from seed 20260921 by scripts/performance_readout.py.