Counterfactuals, Estimands, and Saying What You Mean
Before choosing a method, state what you are trying to estimate: which units, which treatment, which outcome, over what horizon. One campaign produces six different numbers, and only one of them is wrong.
Two people report the effect of the same campaign. One says it lifted conversion by seven points. The other says one point. Both pulled from the same tables, and somebody is now looking for the bug.
There is usually no bug. They estimated different quantities. Most of the time neither of them wrote down which quantity they meant, so nobody noticed until the numbers collided in a meeting.
The thing that is never in the data
Every causal question compares two worlds. The one where the campaign ran, and the one where it did not. Only one of them happened.
That absent world is the counterfactual, and no query returns it. Every method on this site is a different device for constructing a credible stand-in: a randomized holdout, a matched geography, a pre-period, a model. The methods differ in how believable the stand-in is and in what they assume to get there.
None of that machinery helps if nobody has said which comparison they wanted.
An estimand is four choices
An estimand is a precise statement of what you are trying to measure. It has four parts, and each one changes the answer.
Units. Everyone eligible, everyone the message reached, one segment, one region, new customers only.
Treatment. The campaign as sent, the campaign as received, a 20% budget increase, switching the channel off entirely. These are different interventions and they have different effects.
Outcome. Conversions, revenue, retained subscribers at ninety days, margin.
Horizon. Two weeks, one quarter, a year.
Naming all four takes a sentence. Skipping it is why two competent people produce numbers that differ by a factor of five.
The same campaign, read several ways
Show the data
| Percentage points | Point | 95% CI |
|---|---|---|
| Everyone eligible, 4 weeks | +1.32pp | +1.16pp to +1.49pp |
| High engagement only, 4 weeks | +2.24pp | +1.57pp to +2.91pp |
| Everyone eligible, 2 weeks | +1.01pp | +0.87pp to +1.15pp |
| Everyone eligible, 12 weeks | +1.00pp | +0.79pp to +1.22pp |
Everyone eligible over four weeks gives +1.32pp. This is the effect of running the campaign, and it is what a budget decision needs, because the budget buys the campaign rather than the people it happens to reach.
High engagement only gives +2.24pp. Different units, and a perfectly good question if that segment is who you would target next time. It is not a better version of the first number.
Two weeks gives +1.01pp and twelve weeks gives +1.00pp. The short window catches the effect before it has finished arriving. The long window catches it after some of it has been given back, because a share of the lift was purchases moved forward rather than created. Both are correct about their own horizon, and a campaign that looks better at two weeks than at twelve is telling you something specific rather than being inconsistent. One caution: the four-week interval (+1.16pp to +1.49pp) and the twelve-week one (+0.79pp to +1.22pp) overlap, so the data alone does not separate the two horizons here. The give-back is known because the simulation built it in. On a real program it would take a larger test to establish.
Revenue per person gives $0.57, which cannot go on the chart above because it is not measured in the same units. That is worth noticing rather than working around. The people this campaign converts spend less than the people who were converting anyway, so a conversion lift and a revenue lift are different findings, and reporting one while the decision depends on the other is a common and expensive substitution.
The one that is not an estimand
Comparing people who opened the message against the whole control group gives +7.02pp, which is 5.3 times the honest headline. It is not on the chart above, because everything on that chart is an effect and this is not one. It is also the number most likely to end up on a slide, because it is the most impressive one available and it is trivially easy to compute.
It corresponds to no counterfactual. Opening is caused by the treatment, so the opener group is selected on a response to the campaign while the control group is selected on nothing. There is no question of the form “what would have happened to these people” that this comparison answers, because the comparable control group, the people who would have opened had they been treated, is not identifiable in the data.
Write it down before anyone opens a query editor
The practical discipline is one sentence, agreed before the analysis:
The effect of [treatment] on [outcome] among [units], measured over [horizon].
Filling that in takes a minute and settles the argument that would otherwise happen at the readout. It also usually reveals that two people wanted different sentences, which is much cheaper to discover before the work than after it.
Every worked example on this site states its estimand before it states a method, which is the same discipline applied to writing. Designing a Credible Geo Holdout names units, treatment, outcome, and window in a paragraph, and the design follows from it rather than the other way round.
Context
Best suited for
Any measurement question with more than one stakeholder, which is all of them. Especially useful before commissioning work, when the cost of fixing the question is a conversation.
Data requirements
- Nothing beyond what the analysis already needs. This is a specification step rather than a data one
- Assignment recorded separately from delivery and from engagement, so that unit definitions can be applied honestly
- Outcomes available at more than one horizon, if the effect is expected to move
What changes by context
- B2B SaaS
- The unit question is the hard one, because the treatment reaches a person and the outcome belongs to an account. Stating which one the estimand is about resolves most arguments about pipeline attribution before they start.
- Product-led
- Horizon dominates. Activation effects show up in days and retention effects in months, and a single number without a stated window is almost meaningless when both are in play.
- B2C subscription
- Outcome choice is where money is lost. A campaign that lifts trials and not paid conversions is a real result, and reporting the trial number as the effect is the substitution this lesson is about.
When it breaks down
- Exploratory work with no decision attached, where fixing an estimand in advance costs more than it saves.
- Situations where the decision depends on several estimands at once, in which case report several rather than picking a favorite.
- Cases where the treatment itself is ill-defined, such as "brand investment", which needs a definition before an estimand is even expressible.
What this does not establish
- That the honest headline is the right number for every purpose. It answers the budget question. A targeting question needs the segment estimand, and a finance question needs the revenue one.
- That effects always shrink at longer horizons. They shrink here because part of the lift was pulled forward. Effects that compound, such as retention, move the other way, and the direction is a property of the mechanism rather than a rule.
- That the size of the opener bias transfers. Five times the true effect is a property of these open rates and this population. The direction is general; the magnitude is not.
- That naming an estimand makes an estimate credible. It makes the question precise. Whether the answer is believable depends on the design, which is what the rest of this track is about.
- That this simulation varies all four parts. Units, outcome, and horizon vary here. Treatment is held fixed, and varying it is a design question rather than an analysis one.
What to test next
Take the last campaign readout your organization produced and try to write its estimand as the sentence above. If the units are ambiguous, or the horizon is whatever the dashboard defaulted to, or the population was filtered on something that happened after treatment, you have found the thing to fix, and it costs nothing to fix on the next one.
Then check the filters specifically. Any analysis restricted to people who opened, clicked, viewed, or engaged is reporting the number in the last section, and it will be several times too large in the same direction every time.
How the six numbers were produced Code
One randomized campaign over 300,000 eligible people, split evenly, across three engagement tiers with different reach rates, open rates, baseline conversion, and true effects. Reach and opening are recorded separately from assignment, so each estimand can be computed from the same run rather than from a separate simulation.
The true effect on an assigned person at four weeks is 1.38 percentage points, and the headline estimate of +1.32pp recovers it. Effects accrue and partly reverse across the horizons, since a share of the lift is purchases moved forward. Incremental buyers are worth $45.00 against $90.00 for people who would have bought anyway, which is what separates the conversion result from the revenue one.
Intervals are normal approximations on the difference between arms. These are not measurements from a real company.