Triangulating When Your Evidence Disagrees

Three numbers arrive for the same channel in the same quarter. Attribution says one thing, the mix model says another, and the geo test says a third. They are not close.

The usual responses are to average them, to take the one that agrees with the plan, or to send everyone away to reconcile. All three discard the most useful thing in the room: the pattern of disagreement tells you what is happening.

All three methods, one advertiser

Below is the same simulated company the rest of the allocation track uses, with all three methods run on it and the truth reported alongside. That comparison is almost never available in practice, so the reconciliation is usually done on intuition.

Incremental conversions a year, as a multiple of the truth

Channel	Truth	Attribution	Mix model	Experiment	Average of three
Branded search	9,790	5.42x	2.54x	0.85x0.71x to 0.98x	2.93x
Non-brand search	12,606	1.78x	0.70x	0.93x0.83x to 1.03x	1.14x
Paid social	13,036	0.40x	2.03x	0.91x0.82x to 1.01x	1.12x
Online video	9,139	0.15x	0.74x	1.08x0.92x to 1.24x	0.66x
Affiliate	5,308	1.32x	0.35x	1.11x0.83x to 1.38x	0.93x

Averaging can be worse than picking

Look at branded search. Attribution credits 53,029 conversions a year, the mix model 24,822, and the experiment 8,305. The truth is 9,790.

Average the three and you get 28,718, which is 2.93x the truth. Use the experiment alone and you get 0.85x.

The order to read them in

Rank by what each estimate had to assume, not by how precise it looks. Precision runs backwards here: attribution reports an exact number with no interval at all, and it is the least identified thing on the page.

The experiment, for the channel it tested. It withheld the channel and watched what happened, so it needs the holdout to be comparable and nothing else. Its interval is real, and real is not the same as always right: here it covers the truth on 4 channels of five and misses on branded search. That is what a 95% interval does, and it is the only method on the page that can be wrong in a way it told you about in advance. Its scope is narrow: one channel, one window, the spend level that was running.

The mix model, for allocating across channels. It covers every channel at once, including the ones with no click, and it is the only method here that has an opinion about what the next dollar buys. It assumes the variation in spend was unrelated to unobserved demand. That is usually false to some degree.

Attribution, for pacing and deduplication. It describes how conversions arrived. It is fast, complete, and available daily. Its total equals observed conversions by construction, so it is answering a different question from the other two whether or not anyone says so.

The disagreement is the diagnostic

The gaps have signatures. Reading them is faster than re-running anything.

Attribution far above the experiment means demand capture. Branded search is 5.42x of the truth under attribution and 0.85x under the experiment. The channel sits close to the purchase and collects credit for demand that already existed.

Attribution far below the experiment means the channel works upstream where no click is recorded. Online video is 0.15x under attribution and 1.08x under the experiment. A channel that never appears in the path gets nothing from a path-based rule, whatever it did.

The mix model above the experiment means confounded spend. The model has attributed to the channel some of the demand that caused the spending. That is the failure the allocation track works through in detail.

Two of three agreeing is worth more than it looks, and on this advertiser it is the closest thing to agreement available. No channel here produces a three-way match: the tightest is non-brand search, where the estimates still span a factor of 2.53x.

Affiliate is the instructive one. Attribution and the experiment both land at 1.32x while the mix model says 0.35x. Two methods that fail in completely different ways arriving at the same place is real evidence, and it identifies the mix model as the outlier rather than the majority. The reason is known: a model that cannot see a channel recommends switching it off, which is what the worked example flags about this channel.

What to do

Use each for what it can carry. You do not need one number, and manufacturing one is what causes the damage.

Take the experiment as the answer for the channel it tested, and use it to calibrate the mix model rather than to overrule it, which is what calibrating an MMM to experiments is for. Take the calibrated model for allocation across channels. Take attribution for pacing inside a period and for stopping two platforms claiming the same conversion.

Then use the residual disagreement to choose the next experiment. The channel where the methods differ most, weighted by how much money is on it, is the channel worth testing next. On this advertiser that is branded search by a distance, and it is also the largest line in the account.

Context

Best suited for

Organizations past the size where more than one measurement approach is running, and currently reconciling the outputs by argument.

Data requirements

• Each estimate written as an explicit estimand, since most apparent disagreements dissolve at this step
• An interval on anything that has one, and a stated absence where there is not
• Spend by channel over the same window for all three, so the comparison is like for like
• At least one experiment, because without it there is nothing to rank the others against

What changes by context

B2C subscription

Long payback windows mean the three methods often use different horizons, so the first reconciliation step resolves more of the gap here than anywhere. Align windows before concluding anything.

B2C ecommerce

Branded search dominates the disagreement, because it is large, close to purchase, and bid against demand. Expect the biggest gap there and plan the experiment budget around it.

Marketing-led

Enough channels have no clickable touch that attribution is structurally silent on a large share of spend. Treat its silence as missing rather than as zero, which is the error that quietly defunds upper funnel.

When it breaks down

• Only one method available, where there is nothing to triangulate and the move is to state the single assumption you are relying on.
• Methods run on different windows, populations, or outcome definitions, where the disagreement is definitional and reconciling the numbers is the wrong task.
• Experiments too underpowered to rank anything, in which case the hierarchy collapses and the most identified estimate is also the least informative.
• Organizations that need a single number for a target. This lesson argues against producing one. That is a political problem as much as a technical one.

What this does not establish

• That the experiment is always closest. It is here because it was randomized and adequately powered, and even so its interval misses the truth on one channel of five. A contaminated or underpowered holdout can be worse than a decent mix model, and nothing about being an experiment prevents that.
• That averaging is always wrong. It is wrong when errors are correlated, which they are here because two of the three methods share a bias on the same channels. Where errors are independent, combining estimates helps.
• That these ratios transfer. Attribution running at five times the truth on branded search is a property of how much of this advertiser is demand capture.
• That attribution is useless. It is the only method here that is available daily and complete across every tracked touch, which are real properties that the other two do not have.
• That triangulating produces a better estimate. It produces a better understanding of which estimate to use and what the others are telling you about it.

What to test next

Put the three numbers you already have for one channel side by side, with each written as an estimand rather than a metric name. A surprising share of disagreements resolve at that step, and the ones that survive are worth the argument.

Then test the channel with the largest disagreement weighted by spend. That is the one where being wrong costs most, and the disagreement itself is evidence that none of your current methods is settling it.

How the three estimates were produced Code

The same simulated advertiser, fitting code, and geo holdout as the experiment-calibrated mix model, imported rather than re-simulated, so this describes the same company as the rest of the allocation track. The mix model is the history-only fit with no experimental information in it.

The experiment is a 6-week holdout run separately for each channel across 40 markets split at random, estimated by difference in differences on logs. The log difference gives the channel’s share of the conversions observed with it running, and that share is applied to the annual total to put it on the same scale as the other two methods. That step assumes the channel’s share is roughly constant across the year, which for a channel with a seasonal spend spike it is not, and it is an assumption the real version of this comparison has to make too. Branded search comes back at 8,305 with an interval from 6,957 to 9,630, against a truth of 9,790, which sits just above it.

Attribution is modeled rather than simulated from paths: last-touch credit is assigned in proportion to each channel’s spend weighted by how close it sits to the purchase, and the credited total equals observed conversions by construction. The multi-touch lesson simulates journeys directly and finds the same pattern, so the credit rule is an input here rather than the subject. These are not measurements from a real company.