Incrementality · Estimate
Did marketing change the outcome?
What would have happened without the intervention? Counterfactuals, experiments, quasi-experiments, and the diagnostics that decide whether an estimate deserves to be believed. Causal ML sits here too, where the question moves from an average effect to whose effect differs.
What this discipline produces
The same six channels, measured against holdouts rather than credited by a rule. Each answer is a share of what the channel was credited with, and each one carries an interval, because a number without one cannot tell you whether the difference is real.
Worked examples
Full analyses in this discipline, carried from the business decision through to a recommendation and an explicit account of what the result does not establish.
Lessons
Counterfactuals, Estimands, and Saying What You Mean
Before choosing a method, state what you are trying to estimate: which units, which treatment, which outcome, over what horizon. One campaign produces six different numbers, and only one of them is wrong.
Choosing Between User, Geo, and Cluster Holdouts
The design decision is about what you can withhold cleanly, not about statistical efficiency. A procedure for picking the unit, in which power is the last question rather than the first.
Where Precision Comes From in a Holdout
Pairing, covariate adjustment, a weighted control, and randomization inference, applied to one geo test in two noise worlds. Most of the levers people reach for buy nothing in a geo test. The design and the inference are what move the interval.
Prediction Is Not Persuasion
A model that knows who will convert does not know who you can convince. The two questions have different answers, and only one of them can be checked against data you already have.
When Randomization Is Impossible
Difference-in-differences, synthetic control, and interrupted time series. What each one assumes, the diagnostic that would falsify it, and how far wrong the same estimator goes when the assumption fails.
When the Rollout Was Staggered
Markets adopt in waves and the effect grows after adoption. The regression everyone runs on that panel uses early adopters as controls for late ones and understates the effect by a third. What to run instead, and the plot that shows whether it can be believed.
The Diagnostics That Earn Belief
Balance, overlap, pre-trends, placebo tests, and sensitivity analysis. Which to run, in what order, and the thing they all share: they can falsify a design and never confirm one.
When the Experiment Leaks
Noncompliance, contamination, and spillover are three different leaks that bias in different directions. Intention to treat survives two of them, and the third is only fixed by the design.
Triangulating When Your Evidence Disagrees
An experiment, a mix model, and attribution give three different answers for the same channel. A structured way to reconcile them, and why averaging can be worse than picking the best one.
The machinery behind this
Lessons say how to answer the question once. These are the standing systems that answer it on a schedule, each built around the unit it is allowed to withhold.
- Withhold: the person or account Lifecycle Measurement Which of our owned-channel campaigns are worth running, and for whom?
- Withhold: the eligible visitor Growth Experiments Did the change to signup, onboarding, or pricing create value that lasted, and what did it cost downstream?
- Withhold: the market Performance Marketing What is each channel contributing, and where should the next dollar go?
- Withhold: the page Organic Search Did the changes we shipped earn this traffic, or would it have arrived anyway?
- Withhold: the account or territory Sales-Led Pipeline Did this program create, accelerate, or expand pipeline beyond what the sales motion would have produced anyway?