The Diagnostics That Earn Belief

The estimate is a 6% lift with an interval that excludes zero. The slide says so, and the slide is correct about the arithmetic.

Whether anybody should believe it depends on a set of checks that were either run and not reported, or not run. The estimate looks identical in both cases, because a point estimate and an interval carry no information about whether the design that produced them was sound.

None of these checks can tell you a design is valid. Every one of them can tell you it is not.

That asymmetry is not a limitation to be worked around; it is what makes them worth running. A design that passes every diagnostic has survived the tests available. That is a much weaker statement than validated and a much stronger one than unexamined. Reporting them is how a reader distinguishes an estimate somebody stress-tested from one that came out of a query.

Five checks, what each rules out, and what it cannot touch
Check	Rules out	Cannot rule out
Balance	A broken randomization or a pipeline that dropped units unevenly	Confounding by anything unobserved, which is the whole problem in observational work
Overlap	Comparing units that have no counterpart in the other arm	That the units which do overlap are comparable on what you did not measure
Pre-trends	Groups that were already diverging before treatment	That they would have continued in parallel, which is the assumption and is untestable
Placebo	An estimator that finds effects where none exist	A confounder that arrives at the same time as the treatment
Sensitivity	A conclusion that a small amount of hidden bias would overturn	The existence of that bias, which it assumes rather than detects

The third column is the one worth reading. Each check has a specific blind spot, and the blind spots do not overlap, so the set is run rather than the favorite.

The order is by what can kill the design

Balance first, because it is cheap and it catches the failure that invalidates everything downstream. In a randomized design, arms should match on pre-period outcomes and a handful of covariates. They will not match exactly and they should not be expected to; what matters is a difference large enough to indicate that assignment or the pipeline did something. This is the check that catches an experiment assigned at send time rather than decision time.

Overlap second, in any observational design. If a covariate value appears only among treated units, there is nothing to compare them against and the model will extrapolate silently. Trimming to the region of common support is the fix, and it changes the estimand to the effect among comparable units, which has to be said out loud.

Pre-trends third, in anything difference-based. Plot both groups over the pre-period rather than testing a coefficient, because a test that fails to reject on a short noisy window is not evidence of parallelism.

Placebo fourth. Run the whole estimator on a period where nothing happened, or on an outcome the treatment could not affect. If it finds something, the design manufactures effects and the real estimate is not interpretable.

Sensitivity last, because it assumes the design survived the others. It answers how strong an unmeasured confounder would have to be to explain the result away, which converts an argument about whether confounding exists into an argument about how large it would have to be.

Report them when they pass, or they mean nothing when you do

The value of a diagnostic depends entirely on it being run before the result was known and reported regardless of the outcome. Checks run only when a result looks surprising are a filter, not a diagnostic: they remove the surprising findings and leave the comfortable ones unexamined.

The practical form is a fixed block in every readout, with the same checks every time, present whether or not anybody asked. That also makes the missing ones visible.

What a sensitivity analysis reports Technical

The question it answers is how strong an unmeasured confounder would need to be, in its association with both treatment and outcome, to move the estimate to zero. The output is a threshold rather than a probability, and its use is comparative: a confounder would have to be twice as predictive as anything you measured, a claim a reader can evaluate against their knowledge of the setting.

That framing is far more useful than the argument it replaces. “Have you controlled for everything?” has no answer. “Could there be something twice as strong as prior purchase behavior that you did not measure?” is a question somebody who knows the business can weigh, and can sometimes settle.

Two limits. It assumes a single confounder of a specified form, so several weaker ones acting together can exceed the threshold without any one of them reaching it. And it says nothing about whether such a confounder exists; it prices the hypothesis rather than testing it. Reported as a threshold with that caveat, it is one of the most useful lines in an observational readout. Reported as a probability that the result is real, it is a fabrication.

Context

Best suited for

Any readout that will be used to move money, and particularly observational estimates where the design cannot defend itself and the diagnostics are most of what a reader has to go on.

Data requirements

Pre-period outcomes for both arms, long enough to see a trend rather than a fluctuation
Covariates recorded as at assignment rather than as at readout
A period or an outcome where no effect is possible, to serve as the placebo
The assignment mechanism written down, since balance is interpreted differently depending on it

What changes by context

B2C ecommerce: Promotional calendars provide both the best placebo periods and the worst confounders, since a promotion landing with a test is the confounder that arrives at the same moment and passes every check.
Marketplace: Interference means balance and pre-trends can both pass on a design that is still contaminated, because the problem is between arms rather than within them. The relevant diagnostic is a spillover gradient, which none of the standard five is.
B2B SaaS: Small samples make every check underpowered, so a passing balance test is close to uninformative. Plotting the raw pre-period series is more use than any test statistic computed on a dozen units.

When it breaks down

Designs with no pre-period, where the most informative checks are unavailable and the estimate rests on assignment alone.
Confounders that coincide with the treatment date, which fail no diagnostic in this set and can only be handled by knowing the setting.
Readouts where the checks are run after the result is known, in which case they are a filter on which results get questioned rather than a diagnostic.

What this does not establish

That passing these checks makes an estimate causal. They test necessary conditions; a design can pass all five and still be confounded by something nobody measured.
That a failed balance test invalidates a randomized experiment. With enough covariates some will differ by chance, and the question is whether the difference is large enough to indicate a mechanism rather than whether any difference exists.
That trimming to common support fixes an observational design. It removes units with no counterpart and changes the estimand to the region that remains, which is a narrower question than the one asked.
That a sensitivity threshold is a probability. It prices a hypothesis about hidden bias and does not estimate whether that bias is present.