Exposure and Decision Logging

The readout is due and the campaign ran. A policy chose an offer for every customer in the treatment arm, the arm converted better than the holdout, and the deck almost writes itself.

Then someone asks two questions. Which customers saw the offer, given that a third of the sends bounced, went to spam, or landed on a person who never opened the app that week. And what would the policy have chosen for the people in the holdout, since knowing that is the only way to tell whether the policy is good or whether the offer is good.

Neither question has an answer. Not because the analysis was done badly, but because the two facts needed to answer them were never written down, and neither can be reconstructed afterwards at any price.

The decision it encodes

What did we decide for this customer, why, and did it reach them?

Most logging answers a narrower question, which is what got sent. That is enough to bill a vendor and not enough to measure anything.

Sent is not seen

Between a decision and a human being there is a chain, and every link drops people who are not a random sample of the people before them.

Treated arm still in the chain
What that group converts at with no campaign

Show the data

Stage	Remaining	Share of assigned	Baseline rate	Who is lost
Decided	149,948	100.0%	5.2%	Nobody lost yet. This is the population the randomization applies to.
Not suppressed	137,882	92.0%	5.4%	Opt-outs, complaints, and frequency caps already spent.
Sent	135,351	90.3%	5.4%	Records the campaign job could not process, which are rarely typical records.
Accepted	123,022	82.0%	5.7%	Hard bounces and platform rejections, skewed toward stale addresses.
Delivered	80,613	53.8%	6.7%	Provider filtering, which keys off prior engagement directly.
Opened	31,708	21.1%	9.1%	Whether the person looked. Caused by the treatment and by engagement both.

The bars are an operations problem. The dots are the measurement problem: the people who opened convert at 9.1% with no campaign at all, against 5.2% for the holdout they would be compared against. None of that gap is the campaign. All of it is who survived the chain. Simulated data · scripts/exposure_funnel.py

This chain is why the analysis rule is narrower than it looks. Compare the arms as they were assigned, including everyone who was suppressed, bounced, or never opened. That is intention to treat, it is the comparison the randomization supports, and it answers the question a budget owner is asking, which is what happens if we run this program.

The size of that filter is worth seeing rather than asserting. The same simulated campaign, reported three ways:

One campaign, three reported effects, and what the simulation knows to be true
Reported as	Effect	95% CI	What it is
Openers against the whole holdout	+6.83pp	6.45 to 7.20	Not an estimand. Opening is caused by the treatment, so this compares a selected slice of one arm against the whole of the other.
As assigned, everyone	+1.36pp	1.19 to 1.53	Intention to treat. The effect of running the campaign, which is the decision a budget owner is making.
Among those reached	+2.53pp	2.21 to 2.84	The intention-to-treat effect divided by the exposure rate. Valid because the holdout's exposure rate is zero and the campaign cannot act on someone it never reached.

The generating process put the true effect at +1.38pp across everyone assigned and +2.56pp among those reached, so the second and third rows recover what is there and the first overstates the program by a factor of five. The first is also the one that gets presented, because it is the only one that needs no holdout arithmetic and it is by far the most flattering.

Exposure still deserves to be logged, precisely and separately. It is what turns a disappointing intention-to-treat result into a diagnosis: an effect that looks weak because half the arm was never reached is a delivery problem, and an effect that looks weak among people who were definitely reached is a creative problem. Those get fixed by different teams.

Log the decision as well as the action

The second unanswerable question is the one that matters more every year. When a policy or an agent chooses among candidate actions, the chosen action is the smallest part of what needs recording.

A decision record has to carry the alternatives, the scores, and the reason. Without those, the log tells you what happened and supports no counterfactual at all, so the only way to evaluate a new policy is to ship it and wait.

Field	Why it exists
Decision id	Joins the decision to every downstream delivery event
Decision timestamp	Distinct from send time, and the moment the arm applies from
Unit and arm	Which assignment this decision was made under
Policy version	Policies change weekly. Old results must stay interpretable
Candidate set	What the policy could have chosen, not only what it did
Scores	The estimate behind each candidate, so ranking is reconstructable
Action probability	The chance this action had of being chosen
Reason code	Which rule or constraint decided it, including suppressions
Audience version	Ties the decision to the population definition in force

The action probability is the field teams skip and the one that buys the most. If a policy sometimes explores, and its chance of choosing each action is written down at the time, then later you can estimate what a different policy would have earned on the same traffic by reweighting. If it is not written down, that estimate is unavailable forever, because the probability was a property of a model version that no longer exists.

This is the component that closes the loop

Everything else in this section is build order. This is the one that makes the thing run continuously.

The measurement layer reads these logs to produce effects. The decisioning layer reads those effects to choose actions. Those actions are decisions, which land back here, and become the evidence the next measurement is built on. Skip this component and the chain is a pipeline that runs once, because nothing downstream leaves a trace anything upstream can read.

Maturity, honestly staged

Stage 1: delivery outcomes recorded per stage. Sent, accepted, bounced, delivered, suppressed with a reason. No modeling yet, and it already separates delivery failures from creative failures.

Stage 2: decisions logged separately from sends. Decision id, timestamp, arm, and reason code, joined to the delivery chain. Intention-to-treat analysis becomes possible without reconstruction.

Stage 3: candidate sets and scores. The policy’s alternatives and its ranking are recorded, so a readout can say what was nearly chosen.

Stage 4: action probabilities. Off-policy evaluation becomes available, and the organization can test policies against history rather than only against next quarter.

Intention to treat, compliance, and reweighting Technical

Intention to treat compares arms as assigned. It is unbiased under randomization and it estimates the effect of the decision to run the campaign, which is the decision a budget owner is making.

When exposure is partial, a second estimate is available without conditioning on exposure, and its label matters. Dividing the intention-to-treat estimate by the difference in exposure rates between arms estimates the effect among customers whose exposure the assignment changed, the complier effect, and it does so only under random assignment, a real first stage, no effect of assignment except through the defined exposure, and one-sided compliance. It is not the observed effect among everyone who happened to be reached, and it should be labeled as the assumption-dependent estimate it is: “effect among those whose exposure the assignment changed” rather than “effect on those reached”. In a holdout design the control arm’s exposure rate is zero, so the denominator collapses to the treated arm’s exposure rate. Its interval comes from the Wald ratio or a bootstrap, not from dividing the intention-to-treat interval by a denominator treated as fixed, and it is suppressed when the first stage is weak or exposure is measured by opens, which the assumption below rules out as an exposure signal.

ITT   = Y_treated − Y_control
CACE  = ITT / (exposed_rate_treated − exposed_rate_control)

Off-policy evaluation uses the logged probabilities directly. If the old policy chose action a for a customer with probability p, and a candidate policy would choose that same action with probability q, then reweighting the observed reward by q / p estimates what the candidate would have earned. The estimator has high variance wherever p was small, which is the practical argument for a policy that keeps some exploration rather than always taking its best guess.

The raw ratio is rarely what gets used. Clipping the weights at a ceiling, or normalizing them by their own sum, trades a little bias for a large reduction in variance and is what makes the estimate usable on real logs. Pairing the reweighting with a model of the reward, the doubly robust form, stays consistent if either the reward model or the logged probabilities are right rather than requiring both. A team that logged its probabilities can choose between all three. A team that did not has none of them, and no amount of later sophistication recovers the option.

Context

Best suited for

Any program where decisions are made per customer by a model or a rule set, and especially where a policy is expected to improve over time rather than ship once.

Data requirements

A decision event emitted at decision time, independent of whether anything was later sent
Delivery outcomes from the channel platform, per stage rather than collapsed into one flag
A stable decision id that survives the join between the decisioning system and the channel
Policy version and, where the policy is stochastic, the probability each action carried

What changes by context

Lifecycle-led: Email delivery is the whole problem here. Provider-side filtering keys off prior engagement, which is also the strongest predictor of conversion, so exposure and outcome share a cause and exposure can never be used as a filter.
Product-led: Exposure is more observable, because the surface is yours and a render is a real event. The harder part is that the decision often happens client-side, so the log has to survive an unreliable network without dropping the records it could not send.
B2C subscription: Decisions accumulate against the same person for years, so policy version history matters more than elsewhere. A readout that cannot say which policy version was live in a given month cannot explain its own trend.
B2B SaaS: The decision is often made about an account and delivered to a person, so the log has to carry both grains. Collapsing to one loses either the assignment or the exposure.

When it breaks down

Channels that report no delivery detail, where exposure is a vendor aggregate and cannot be joined to a person.
Policies that are deterministic and never explore. Nothing can be evaluated off-policy, because no alternative was ever taken.
Systems where the decision and the send are the same event, so there is no way to separate a delivery failure from a decision not to act.
Any pipeline that overwrites decisions on retry. A rerun that replaces the original record destroys the thing the log existed to preserve.