Context · Business model
Local / omnichannel
Physical locations, offline conversion, and geography as the natural unit.
What changes here
The same four questions, and what each one runs into in this context.
Attribution
Online-to-offline linkage is weak, and most of the conversion is invisible to any tracking you own.
Incrementality
The strongest case for geo designs, because the market is the unit. Outcome measurement is the hard part, not assignment.
Allocation
Store capacity and catchment area cap what any budget can produce, whatever the response curve says.
Decisioning
Local inventory and staffing constrain the action set, so a policy that ignores them is unexecutable.
Written for this context
Incrementality
-
When Randomization Is Impossible
Difference-in-differences, synthetic control, and interrupted time series. What each one assumes, the diagnostic that would falsify it, and how far wrong the same estimator goes when the assumption fails.
-
Where Precision Comes From in a Holdout
Pairing, covariate adjustment, a weighted control, and randomization inference, applied to one geo test in two noise worlds. Most of the levers people reach for buy nothing in a geo test. The design and the inference are what move the interval.
-
When the Rollout Was Staggered
Markets adopt in waves and the effect grows after adoption. The regression everyone runs on that panel uses early adopters as controls for late ones and understates the effect by a third. What to run instead, and the plot that shows whether it can be believed.
- Worked example
Designing a Credible Geo Holdout
A paid social incrementality test carried end to end: market pairing, power, pre-trend diagnostics, a difference-in-differences estimate with its interval, incremental CAC, and what the design cannot settle.
Systems
- System
Geo Design and Market Matching
Choosing which markets go dark, for how long, and against what, so that the difference afterwards means something. The sample size is the number of markets, not the number of people in them.
- System
The Persistent Geo Holdout
A small, rotating share of markets kept dark on the largest channel, so calibration has a recurring anchor without a negotiation. The standing control the calendar cannot provide, when it is worth running, and the price of having one.
- System
Budget Reallocation
Turning a set of curves into a plan somebody will run, under the contracts and minimums that make the unconstrained optimum unreachable, and choosing the moves that generate the next experiment.
- System
Content Inventory and Template Map
Every URL mapped to the template that renders it, the intent it serves, and whether it is indexed at all. The unit of change and the unit of randomization are not the same thing, and both have to be known.
- System
Template Split Design
The only control organic gets is the other half of a template, so the design question is which pages take the change, stratified by the traffic concentration that decides whether it can detect anything.
- System
Paid and Organic Together
Some of an organic gain on a term you also buy is your own paid traffic arriving free, and reporting the two channels separately can count the same click twice. How much is measured, not assumed, and who owns the number.