Evidence Review

The readout is automated. The decomposition runs on a schedule. Both landed in a dashboard on Monday, and by Thursday the planning deck quotes whichever number was largest.

Automating measurement removes the delay and the arithmetic. It does not decide which estimate a decision should rest on, whether the evidence clears the bar for acting at all, or what happens when two sources disagree. Somebody does that, and in most organizations nobody is named.

The decision it encodes

Which estimate governs this decision, and is it strong enough to act on?

This is the only component here whose output is a judgment rather than a number, which is why it needs to be a component. A judgment that is made informally is made differently every time, by whoever is in the room.

Rank by assumption

The estimates arriving here differ in how much they had to assume, and precision runs close to backwards against that. An attributed figure is exact and rests on a credit rule with no counterfactual in it. A holdout estimate carries a wide interval and rests on randomization.

The ranking is the starting prior, and it is not the whole judgment. An old randomized test of a different offer on a different population should not govern over a recent model calibrated directly to randomized evidence for the action in question. Each estimate is scored on six things before one is promoted.

What each estimate is scored on

Dimension	The question
Identification	Is there a credible counterfactual, and what did it assume?
Estimand match	Same treatment, population, outcome, and horizon as the decision in front of us?
Execution	Did assignment, exposure, and measurement pass their checks?
Transport	Does it apply to the current context, or to one that has since changed?
Precision	Is the interval narrow enough that the decision differs across it?
Materiality	Would plausible values inside the interval change what we do?

Where two sources disagree, the disagreement is information before it is a problem. Its direction and size say which mechanism is operating, which is faster than re-running anything. Triangulating When Your Evidence Disagrees works through the readings, including why averaging conflicting estimates can be worse than using the best one alone.

Inconclusive has to be a permitted verdict

The readout can already report that a test was not powered to detect an effect worth acting on. That verdict only means anything if something downstream is willing to accept it.

An organization where every review ends in a decision will convert weak evidence into confident recommendations at the rate weak evidence arrives. The useful discipline is that “we did not learn anything, here is what it would take to” is a normal outcome that closes the item, rather than a failure that sends someone back to find a better cut of the data.

What it produces

Not a report. Three things that other components consume.

A governing estimate per decision, named and dated, with its source and its interval. This is the number the plan uses, and the point of promoting it explicitly is that everyone is then arguing about the same one.

A verdict, which may be inconclusive. Where it is, the review states what would resolve it, and that becomes the next experiment.

A queue, ordered by expected value of information: the stakes of the decision, times the chance that better evidence would change it, less the cost and delay of getting it. This is the input to the experiment roadmap, and it is the most valuable output here because it is the only thing that turns disagreement into a plan.

Maturity, honestly staged

Stage 1: a named owner and a standing slot. Someone is responsible for saying which number governs. Most of the value is here and it requires no tooling.

Stage 2: estimands attached to estimates. Each figure arrives with its population, outcome, and window, so definitional disagreements get separated from evidential ones before anyone argues.

Stage 3: a recorded governing estimate. The chosen number is written down with its source, its interval, and its date, and downstream systems read it rather than a spreadsheet.

Stage 4: the queue drives the roadmap. Experiments get scheduled by how much spend rests on an estimate nobody can defend, rather than by whoever asks.

Why this is a component rather than a meeting Technical

Because it has inputs, an output other systems consume, and a decision rule. The governing estimate is read by the next-best-action policy, which needs an incremental number to optimize against and should not be picking one for itself. The queue is read by whoever plans experiments. The verdict closes or reopens an item.

Calling it a meeting would leave those consumers reading whatever they found. The reason it is drawn here between measurement and decisioning is that the policy downstream is only as good as the number it was pointed at, and something has to be accountable for choosing it.

Context

Best suited for

Organizations with more than one source of measurement evidence, which is any of them running both experiments and models, and currently reconciling the outputs in whichever meeting notices first.

Data requirements

• Estimates arriving with an estimand, an interval, and a named source
• A power verdict from the readout, so inconclusive can be distinguished from null
• Spend or exposure per decision, to order the queue by what is at stake
• Somewhere durable to record the governing estimate that downstream systems can read

What changes by context

B2C subscription

Payback windows are long enough that the governing estimate changes as a cohort matures, so the record needs a horizon attached and a scheduled revisit rather than a single sign-off.

Lifecycle-led

Volume of readouts is the constraint. Most items should close in a line, and the forum earns its time on the handful where sources disagree or the money is large.

B2B SaaS

Sample sizes make inconclusive the usual verdict rather than the exception, so the discipline that matters most here is being willing to record it and move on instead of reaching for a weaker design that returns something.

When it breaks down

• A single source of evidence, where there is nothing to adjudicate and the move is to state the one assumption being relied on.
• Organizations where the governing estimate can be overridden without record, which makes the whole component decorative.
• Decisions that have to be made faster than the forum meets, which need a standing rule agreed in advance rather than a review.
• Cultures that treat an inconclusive verdict as a failure, where this reliably degrades into a search for a number that clears the bar.