One Week of the Lifecycle System's Readout
Four campaigns read out in the same week, as the system would emit them: the checks that run before any lift is read, the effect as assigned and the complier effect beneath it, the guardrails against their bounds, a power verdict, and the decision the review rule produced. One of the four learned nothing, and says so.
This is what lands on the Monday. Not a specification of a readout, the readout: every campaign that reached its four-week horizon in Week 37, against its own holdout, run through the same checks in the same order, with the verdict the review rule produced before anyone had an opinion about it.
Every number is simulated, from four invented campaigns, and every one comes from the script named at the end. What is not invented is the shape, and the shape is the point: the components in this system each specify one column of this table.
The week
| Campaign | Arms | Effect as assigned | Guardrails | Verdict |
|---|---|---|---|---|
| Winback offer, lapsed 90 days | 216k / 23.8kratio p 0.23 · 58% reached | +1.38pp+1.09pp to +1.68pp | Unsubscribes passesComplaints passes | Effect detected |
| Annual plan upsell, month-two | 32.5k / 3.5kratio p 0.30 · 71% reached | +0.94pp−0.05pp to +1.93pp | Unsubscribes passesComplaints passes | Inconclusive |
| Discount nudge, cart abandoners | 162k / 18.0kratio p 0.84 · 66% reached | +1.46pp+1.05pp to +1.86pp | Unsubscribes breachedComplaints passes | Guardrail breached |
| Feature tips, active users | 378k / 41.9kratio p 0.67 · 74% reached | +0.01pp−0.31pp to +0.33pp | Unsubscribes passesComplaints passes | No effect at the size that mattered |
Read left to right, because that is the order the checks run in. The sample ratio and the guardrails are read before the effect, so a campaign with an invalid split or a breached bound is seen as that first, whatever its lift. “Unreadable” on a guardrail means the holdout was too small to rule the bound out, which is a finding about the design and not a pass.
Four verdicts in one week is what a program of this size produces. Two matter for what the reader learns about the system rather than the campaigns: the guardrail that was breached on a campaign whose primary metric may well have moved, and the campaign that learned nothing and was allowed to say so.
One campaign in full
The winback offer, read the way every row above was read.
| Assigned | 240,000 eligible. 216,178 treated, 23,822 held out. Sample ratio against the 10% design: p = 0.23. The arms are the arms the design said. |
|---|---|
| Delivered | 17,124 suppressed before send, 199,054 sent, 171,331 accepted by the provider, 125,640 seen. Exposure rate 58% of the treated arm, which is the number the two effects below differ by. |
| Effect as assigned | +1.38pp at four weeks (+1.09pp to +1.68pp), from 13,881 conversions against 1,200. At two weeks it read +1.15pp (+0.93pp to +1.36pp). This is the effect of running the campaign, on everyone it was decided for, and it is the number the budget conversation uses. |
| Effect among those whose exposure the assignment changed | +2.38pp (+1.87pp to +2.89pp): the effect as assigned divided by the exposure rate, a complier effect that holds under the assumption that the message did nothing to the people it never reached and that exposure was measured as delivery rather than opens. Reported second and labeled as assumption dependent, because it is the diagnosis rather than the decision. |
| Guardrails | Unsubscribes per thousand: −0.02 (−0.69 to +0.65) against a bound of +1.0. Passes.Complaints per thousand: −0.03 (−0.27 to +0.22) against a bound of +0.5. Passes. |
| Power verdict | This design could detect 0.42 points at conventional power. The program asked to see 0.5. Adequately powered: a null here would have meant something. |
| Decision | Effect detected. Continue and scale to the full audience. |
The effect as assigned and the complier effect are both here, in that order, and they differ by the exposure rate. A readout that carried only the second would be quoting an assumption-dependent number about a subset as if it were about the program.
The one that learned nothing
The annual plan upsell went to 36,000 people, which is a large campaign by most standards and a small experiment by this one. Its effect as assigned reads +0.94pp with an interval from −0.05pp to +1.93pp, which contains zero, the effect the program hoped for, and an effect twice that size. Its design could detect 1.4 points; the program wanted to see 0.5.
The verdict is inconclusive, and the readout says so in those words rather than reporting the point estimate with a caveat underneath. That is the single hardest thing to make a measurement system do. The temptation is to call the campaign a mild success or a mild failure, because either is a story and “we did not learn anything” is not. The decision attached to it is the useful one: pool with the next run’s own control, which the registry makes possible by keeping the outcome contract fixed across runs, rather than re-run it at the same size and learn nothing again.
The program, quarter to date
Persistent holdout · 11 weeks into the quarter
50,000 customers withheld from every discretionary action for the quarter, against 4,950,000 who received the program. This is the program holdout, a separate randomization from the four campaign controls above, and none of the four rows was read against it. Conversion is +0.34pp higher in the program arm (+0.16pp to +0.51pp), a relative lift of 8.2% on the quarter’s baseline.
This line does not move when a campaign changes, ships, or is retired. It is what the program is worth, and it is the number the four rows above are eventually reconciled against.
What this does not establish
- That the winback effect persists past four weeks. The two-week read is smaller than the four-week one, and whether the four-week read holds at twelve is a horizon the readout has not reached yet.
- That the discount nudge should never run. It breached a guardrail as designed; a variant with the same offer and a different frequency is a new campaign with its own readout.
- That the upsell campaign did nothing. It was not readable at this size, which is a statement about the design and not about the campaign.
- Anything about a real company. Four campaigns were invented so the readout would need every verdict it has; the shape is the site’s claim, and the numbers are the script’s.
How this readout was produced Code
Four campaigns are simulated with stated sizes, exposure rates, baselines, true effects on the exposed, and true unsubscribe costs, chosen so the week would need every verdict in the rule. Each is randomized with a 10% holdout, delivery is a chain of losses before exposure, outcomes are binomial at four weeks, and every interval is a normal interval on a difference in proportions, the same estimator the exposure component describes.
The sample ratio check is a chi-square on the arm counts against the design share. Guardrails are non-inferiority tests against a stated bound with three states: pass when the interval excludes the bound, breach when a degradation is established and the point sits past the bound, unreadable otherwise. The power verdict is the effect the observed standard error could detect at 80% power against the effect the program said it needed. The rule runs in a fixed order: validity, guardrails, primary, and a verdict of inconclusive when none of those settled it.
Regenerated from seed 20260919 by
scripts/weekly_readout.py. The true effects the script planted
are in the JSON beside the estimates, which is the one thing a real readout
can never show.