Performance Marketing

Three numbers arrive for paid social every quarter: the platform’s, the finance spreadsheet’s, and the mix model’s. They have never agreed, and the plan is built on whichever one the person presenting believes.

The model has been checked against an experiment once. Its number for paid social has moved twice in three years, both times because somebody changed a prior. There is no point in that history where it was told it was wrong by something outside itself.

Most bought media cannot be withheld from a person

Someone who sees your ad on a streaming service, a billboard, or a display network cannot be identified beforehand and excluded. The finest thing you can turn off is a market. Where a channel does support a valid person-level lift design, use the finest credible unit and measure it the way the lifecycle system does; this system is for the rest.

Everything else here follows from that. Sixteen markets is sixteen units, however many customers each one sells to, so precision comes from the number of markets and how alike they are rather than from the size of the audience. That is why a geo test needs a far larger effect before it can see anything than a lifecycle holdout does, and why performance reads four times a year in the figure the section opens with while lifecycle reads fifty.

The four stages, in plain terms

Spend spine is one vocabulary for channels and one spend series everyone agrees on, reconciled to what the business paid rather than to what a platform reported, joined to the outcome the budget is meant to move and the value the business puts on it. It sounds like bookkeeping and it decides which questions are available later, because the grain you keep is the grain you can ask at. Aggregating daily market spend into weekly national totals is irreversible, and it removes the geo design from the menu permanently.

Design is how a comparison gets manufactured. Markets matched on how they have behaved rather than on how they look, randomized within matched pairs, and a standing calendar so experiments exist at all. The calendar is the part that converts this from a study into a system, and it is where the section’s uncomfortable arithmetic lives: a hundred markets and twelve-week cycles gives about four readable tests a year, so which channel occupies a slot is a real decision. A persistent geo holdout on the largest channel adds the one thing a calendar cannot: a comparison that exists every week, so calibration has a live anchor between tests and a channel that drifts is caught before the next slot.

Estimation is response curves and calibration. The curves are where most of the modeling effort goes and identification is the real subject: if spend was set by a rule, high-spend periods are also high-demand periods and the fit will attribute the demand to the media. An experiment anchors the part of a curve where the business created spend variation; the rest is interpolation or extrapolation and is labeled as such. Calibration is where a disagreement between an experiment and the model has to resolve rather than being reported twice.

Allocation turns the curves into a plan somebody will run, under the contracts, capacity limits, and rate-of-change constraints that make the unconstrained optimum unreachable. It also records every move with the effect it predicted, so the following quarter scores this one and the model is trusted on its record rather than by default.

The loop is what makes it compound

The dashed edge is the only one that is not a build dependency. Every reallocation is logged with the curves behind it, the effect it predicted, and what happened. A move whose outcome sat in tension with its forecast names a channel the model was wrong about, and that error, weighted by the money on the channel and by whether the decision would change, moves it up the calendar. A move staged by market first is itself the test, at a spend level the history never covered, which is the variation the curves could not identify from observational data.

The rule that keeps the two apart is short. Staged decisions update the evidence base. Unstaged misses update the test queue. A forecast miss on a move made everywhere at once is diagnostic, because everything else changed in the same quarter, and a system that wrote it into the curves would be correcting the model with noise.

Without that edge the whole thing is an annual model refresh: fit, recommend, ship, and arrive next year with a year more of spend that still followed a rule and still identifies nothing, and a model that has made three recommendations and been checked against none.

What runs on what cadence

The loop closes quarterly, and that is the cadence of the causal updates, not of the system. The instrumentation runs continuously; the curves and the budget move when evidence has matured.

What happens when

Cadence	What runs
Daily	Spend feeds checked for gaps, delivery audited against the holdout and any active test, tracking failures caught while a restart is still cheap.
Weekly	Achieved treatment contrast, contamination, and outcome completeness reviewed for every running test; guardrails read.
Monthly	Platform delivery reconciled to invoiced cost and to outcomes by market; evidence freshness per channel reviewed against its staleness date.
Quarterly	Experiments read, the supported region of each curve updated, the budget reallocated, and every move registered with its forecast before it ships.
Semiannually	Market pairs, value assumptions, the channel taxonomy, and whether the standing holdout is still worth its cost reassessed.

Where to start when none of it exists

First, the spend spine at daily and geo grain. Not because it is interesting, but because it is the only decision on this list that cannot be revisited. Everything else can be built later on data you kept; nothing can be built on data you aggregated away.

Then one geo test on the largest channel. One measured effect is worth more than a model of six channels, and it establishes both the design and the appetite for going dark, which is the harder of the two to get. Write down what it was expected to show before it runs, which is the first entry in the decision log.

Then the calendar, before the model. This is the inversion most programs get wrong. A model built first will be refitted for years without ever being checked, because checking it was never scheduled. A calendar built first guarantees the model has something to be calibrated against on the day it arrives. Once the appetite for dark markets exists, convert the first test into a standing holdout on that channel, so the anchor stays live instead of aging until the next slot.

Then curves, calibration, and allocation. In that order, and the allocation last, because a precisely solved plan on uncalibrated coefficients is exact about the wrong thing.

When the market is a store

Retailers run the same design at a finer grain: matched stores rather than matched markets, hundreds of them, on a standing platform that reads dozens of tests a year. Everything here transfers. The matching is on store history, the estimator is the same difference in differences, the design effect from unequal stores is the same Kish arithmetic, and the calendar’s capacity constraint is looser because there are more units to split. What changes is what gets tested, which is mostly merchandising, layout, and staffing rather than marketing, so the practice belongs to a store operations team with this system’s design underneath it.

Brand outcomes

Awareness, consideration, and the long-run effect of upper-funnel spend are not a separate system. They are this system’s geo design read on a different outcome: a survey panel by market, or sales read over quarters rather than weeks. The design carries; what does not carry is the precision, since a survey of a few hundred respondents per market is noisier than sales, and the horizon, since a brand effect that takes a year to arrive needs a holdout that lasts one. A program that wants brand measurement should add the outcome to this system and accept those two costs, rather than buy a brand lift study with no control it can name.

What this leaves out

Person-level targeting. Some bought media can be addressed to individuals, and where it can, measure it the way the lifecycle system does rather than to fold it into a market-level model that cannot see it.

Creative. Which asset performs is a real question and a different design, closer to a conversion test than to anything here, and it belongs inside the spend spine as a dimension rather than as a stage of its own.

Methods and further reading

The designs here are not original, and the sources are worth reading in the original.

• Vaver and Koehler, Measuring Ad Effectiveness Using Geo Experiments (Google, 2011): the randomized geo design and its difference-in-differences readout.
• Kerman, Wang, and Vaver, Estimating Ad Effectiveness using Geo Experiments in a Time-Based Regression Framework (Google, 2017): the time-based regression variant, and what a pre-period buys.
• Chen and Au, Robust Causal Inference for Incremental Return on Ad Spend with Randomized Paired Geo Experiments (Google, 2019): Trimmed Match, and why paired geo outcomes need a robust estimator.
• Jin, Wang, Sun, Chan, and Koehler, Bayesian Methods for Media Mix Modeling with Carryover and Shape Effects (Google, 2017), and Chan and Perry, Challenges and Opportunities in Media Mix Modeling (Google, 2017): the curve family and the causal assumptions it rests on.
• Callaway and Sant’Anna, Difference-in-Differences with Multiple Time Periods (Journal of Econometrics, 2021): the estimator for a rotating holdout’s pooled read.
• Gneiting and Raftery, Strictly Proper Scoring Rules, Prediction, and Estimation (JASA, 2007): why coverage is not enough to score a forecast, and what is.