The Test Calendar
Experiments as a standing cadence rather than a request, ordered by how much spend rests on a coefficient nobody has validated. The component that makes calibration possible at all.
The decision it encodes
Which experiment runs next, and what is it for?
The mix model is refitted every quarter. It has been refitted eleven times.
It has been calibrated against an experiment once, two years ago, on a channel that has since been restructured twice. Every quarter the model produces a number for every channel, every quarter the plan is built on those numbers, and nothing in the loop has ever checked them against a comparison.
The model is not the problem. The problem is that nothing was scheduled to check it, so checking it stayed a thing somebody would get to.
The decision it encodes
Which experiment runs next, and what is it for?
An experiment program that runs on request produces evidence about whatever somebody argued about most recently. A calendar produces evidence about what carries the most money, which is a different set almost every time.
This is what makes the rest a system
A single geo test is a study. It answers one question, once, and the answer decays. What turns the estimation layer from a model somebody fits into a measurement function is that experiments arrive continuously, so there is always something recent to calibrate against.
That reverses the usual order. Most teams treat experiments as validation to be done if there is time. Here the calendar is a precondition: the calibration step has nothing to consume without it, and an uncalibrated curve is a description of historical spend patterns wearing the clothes of a causal estimate.
Order the queue by money at risk
Every channel carries a coefficient somebody is spending against. The value of testing it is roughly the spend it governs multiplied by how uncertain the estimate is. A large channel everyone agrees about is worth less to test than a mid-sized one where the model and the platform disagree by a factor of two.
| Field | What it decides |
|---|---|
| Channel and question | Which coefficient this is meant to move, stated before the result exists. |
| Spend governed | The money resting on the current estimate. Half the ranking. |
| Current uncertainty | How far the model, the platform, and the last experiment disagree. The other half. |
| Markets claimed | Which markets are committed, so two tests cannot silently overlap on the same geographies. |
| Window | Start, end, and the outcome horizon, which is usually longer than the flight. |
| Staleness date | When this estimate stops counting as current and the channel re-enters the queue. |
The staleness date is the field that turns a list of past tests into a schedule. Without it the queue only ever grows forwards, and channels validated once stay validated forever.
How many tests you can run
Market capacity is a shared budget, in the way contact capacity is a shared budget for lifecycle. The difference is how little of it there is. A lifecycle program splits a list of millions and two campaigns barely notice each other. Two geo tests running at once each get half the markets, and both get weaker for it.
The arithmetic is short. Split M markets across k simultaneous tests and
then in half again for treatment and control, and each test’s detectable effect
comes out proportional to the square root of k. Running two at once costs each
of them about 40% on the smallest effect it can see.
| Tests at once | Markets per arm | Detectable effect | Tests per year |
|---|---|---|---|
| 1 | 50 | 4.5% | 4 |
| 2 | 25 | 6.3% | 8 |
| 3 | 16 | 7.8% | 13 |
| 4 | 12 | 9.0% | 17 |
Illustrative, at a market-level residual variation of 8% and conventional power. The shape is what matters rather than the numbers: you can have 4 tests a year that detect 4.5%, or 17 that detect 9.0%, and there is no arrangement that gives you both.
That is the constraint the queue exists to allocate. A lifecycle program can afford to test whatever anybody is curious about, because a holdout on one more campaign costs almost nothing. Here the budget is a handful of readable tests a year, so which channel occupies a slot is a real decision with a real cost, and ordering the queue by money at risk stops being a refinement and becomes the whole job.
Reserve the capacity in advance
The reason most calendars collapse is that every market is committed to delivering the plan, so no market is ever available to test with. Holding a standing share of markets, or a standing share of the year, converts testing from something that competes with the plan into something the plan is built around.
That reservation is the real cost of this component and it should be stated in those terms: a fixed percentage of the media budget is being spent to learn rather than to deliver. It buys the ability to know whether the other percentage worked. The strongest form of the reservation is a persistent geo holdout on the largest channel, which makes the comparison standing rather than scheduled and takes that channel’s calibration off the calendar entirely.
Maturity, honestly staged
Stage 1: one test scheduled a quarter ahead. Committing to a date before there is an argument is most of it, because a test scheduled in advance survives the week it becomes inconvenient.
Stage 2: the queue ordered by spend at risk. Which test runs stops being decided by who asked, and starts being decided by what it would cost to be wrong.
Stage 3: a staleness policy per channel. Estimates re-enter the queue on a schedule, so the evidence base stays current rather than accumulating.
Stage 4: reserved market capacity and collision checks. Tests stop competing with the plan for markets, and two tests cannot claim the same geographies without somebody noticing.
Ordering the queue Technical
The ranking is a rough expected value of information. For a channel with
spend S and a current estimate whose plausible range spans
Δ in effectiveness, being wrong somewhere in that range
misallocates on the order of S · Δ. Ranking channels by that
product puts the queue in a defensible order, and the precision of the
ranking does not matter much: what matters is that a large channel with a
wide interval outranks a small one with a narrow one, which is what
intuition tends to get backwards.
Two refinements are worth the effort. Weight by whether the decision would change: a channel you would keep funding at either end of its interval is not worth testing however large it is, because the evidence cannot alter an action. And subtract the cost of the test itself, which for a dark-market design is the forgone contribution in the treated markets over the window, and is not small.
The queue this produces is the paid-media analog of the one evidence review keeps for lifecycle, and for the same reason: disagreement is only useful once it has been turned into a plan for resolving it.
Context
Best suited for
Programs where a model is refitted on a cadence and used for planning, and where nobody can currently say when each channel's estimate was last checked against a comparison.
Data requirements
- Spend by channel, to rank the queue by what is at stake
- A record of every past test with its channel, window, estimate, and interval
- Market availability, so a slot can be reserved rather than negotiated each time
- Whatever the model and the platforms currently claim per channel, since the size of the disagreement is half the ranking
What changes by context
- B2C ecommerce
- Read cycles are short, so several tests can run a year and the staleness window can be aggressive. The constraint is the promotional calendar, which occupies the periods when volume is high enough to test on cheaply.
- B2C subscription
- Outcomes take months, so the calendar has to schedule the readout as far ahead as the test. A test that finishes in March and reads in June should be booked as one item, not two, or the readout gets forgotten.
- Marketplace
- Supply and demand tests compete for the same markets and interfere with each other, so collision checking matters earlier here than elsewhere. Two sides testing simultaneously in one market produces two contaminated results.
When it breaks down
- Programs too small to spare any market or any period, where the answer is a single annual test on the largest channel rather than a calendar.
- Organizations where budget is committed a year ahead and cannot be moved, so a slot cannot be reserved and every test is a negotiation.
- Channels that change faster than a test can read, where the estimate is stale before the readout lands and the calendar produces history rather than evidence.