Context · Business model
B2C subscription
Recurring revenue moves the question from acquisition to retained value, and makes pull-forward the main way a good result turns out to be nothing.
What changes here
The same four questions, and what each one runs into in this context.
Attribution
Trial starts are easy to attribute and are not the outcome. Credit assigned at signup says nothing about month three.
Incrementality
Pull-forward looks like lift in a short window and disappears in a long one. The test window has to outlast the natural purchase cycle.
Allocation
Payback period and contribution margin matter more than CAC. A cheap subscriber who churns at month two is a loss.
Decisioning
Churn models find who leaves, not who can be kept. Uplift is the only honest target for retention spend.
Written for this context
Attribution
-
Attribution Is Not Incrementality
Attribution assigns credit for conversions you observed. Incrementality estimates the ones that would not have happened otherwise. The gap between them is where budgets go wrong.
-
Reading Platform Discrepancies Without Losing Your Mind
Two systems report different numbers for the same campaign. Four causes account for nearly all of it, and checking them in the right order turns a week of investigation into an afternoon.
-
What Multi-Touch Attribution Actually Computes
Seven credit rules run on the same journeys, from last touch to Markov chains and Shapley values, against the effects that generated them. What changes between rules, what does not, and why the sophisticated ones fail in the same place.
-
What Tracking Loss Actually Costs You
Consent refusal, identity resolution, and cross-device loss are not one problem. Separating loss that costs precision from loss that costs validity, and why only one of them is fixable with more data.
Incrementality
-
Counterfactuals, Estimands, and Saying What You Mean
Before choosing a method, state what you are trying to estimate: which units, which treatment, which outcome, over what horizon. One campaign produces six different numbers, and only one of them is wrong.
-
Prediction Is Not Persuasion
A model that knows who will convert does not know who you can convince. The two questions have different answers, and only one of them can be checked against data you already have.
-
Triangulating When Your Evidence Disagrees
An experiment, a mix model, and attribution give three different answers for the same channel. A structured way to reconcile them, and why averaging can be worse than picking the best one.
- Worked example
Designing a Credible Geo Holdout
A paid social incrementality test carried end to end: market pairing, power, pre-trend diagnostics, a difference-in-differences estimate with its interval, incremental CAC, and what the design cannot settle.
Allocation
-
Adstock, Saturation, and Honest Response Curves
Lagged effects and diminishing returns are where most mix models smuggle in their conclusions. How to tell a fitted curve from an identified one, and what variation would identify it.
-
Calibrating an MMM to Experiments
Which experiment to run, how to carry its estimate into the model, and how hard to lean on it. Anchoring a channel's level drags its slope down with it, and the slope is what the media plan runs on.
-
Marginal Return, Not Average Return
The channel with the best cost per acquisition is often the worst place to put the next dollar. Averages describe money already spent. Budgets are decisions about money that has not been spent yet.
-
Paid Search and SEO Are Not Independent
Pause branded paid search and most of those conversions come back through organic. The paid channel's cost per acquisition describes a transfer between two of your own channels, not a purchase of new demand.
-
What a Marketing Mix Model Can and Cannot Do
A mix model is a good allocation tool and a weak causal one. Where its identification would have to come from, why a Bayesian version does not create any, and what makes one worth acting on.
- Worked example
An Experiment-Calibrated Marketing Mix Model
A mix model that fits three years of weekly data almost perfectly and gets the channel contributions badly wrong, and what one geo holdout does about it.
Decisioning
-
From Segment to Audience to Treatment Policy
Segments, audiences, scores, assignments, and policies are five different objects. A result is only comparable to a previous one if you can say which of the five changed.
-
The Case for Doing Nothing
Contact fatigue, capacity, and cost mean no-action is a real treatment arm. Why it belongs in every decisioning system by default, and why organizations reliably leave it out.
-
Uplift Modeling in Practice
Where to cut the ranking, how much to withhold, and why the offline number is larger than the money. Estimating heterogeneous treatment effects when you have a randomized holdout, and validating the policy prospectively.
-
Why Agentic Marketing Still Needs Holdouts
Two agents, the same budget and the same population, differing only in what they are scored on. One improves its dashboard by 65% while destroying value, and the one doing the right thing looks worse.
- Worked example
Broad Nurture, Propensity, or Uplift?
One trial-to-paid nurture sequence, four targeting policies, and a randomized holdout. Compared on incremental conversions rather than model accuracy, which is the only comparison that changes the answer.
Systems
- System
Customer State Model
One row per person or account carrying consent, lifecycle stage, eligibility flags, and contact history. The foundation every audience, experiment, and policy reads from.
- System
Audience Builder and Capacity Planning
Audience definitions as versioned, reusable objects instead of one-off SQL, and the capacity check that decides whether a test was ever going to work.
- System
Contact Permission and Suppression
Lawful basis, channel permission, jurisdiction, purpose, global opt-outs, and complaint handling as one shared layer rather than a filter each campaign reimplements. Consent is one basis among several; suppression is a decision of your own.
- System
Assignment and Experiment Registry
Deterministic hash-based assignment, a program holdout with a scope contract, overlap rules, an outcome contract per experiment, and a registry that stops two campaigns from colliding on the same people.
- System
Exposure and Decision Logging
Logging what was decided and what was delivered, not just what was sent. The component that makes everything downstream measurable at all.
- System
Automated Measurement
A standing readout that runs eligibility, assignment, delivery, exposure, outcome, and guardrails on a schedule instead of as a bespoke analysis each time.
- System
Program Lift and Reconciliation
What the program holdout says the lifecycle policy was worth, reconciled against what the tested actions account for on a common value scale, with the residual reported as a residual. The question a per-campaign readout cannot answer.
- System
Evidence Review
The standing forum where estimates get ranked by what they assumed, a number is promoted to the one the plan uses, and a result is allowed to be inconclusive.
- System
Next-Best-Action Policy
Turning scores into decisions under real constraints: capacity, contact fatigue, margin, cooldowns, and the explicit option of doing nothing.
- System
Treatment Effects and Policy Evaluation
The bridge between a campaign's average effect and a per-customer decision: estimating how the effect differs by customer state, and scoring a candidate policy on randomized or logged data before it is trusted with volume. The component that produces the numbers next best action consumes.
- System
Policy Governance and Safety
Who may change the policy, what it may do, what stops it without a human, and how anyone knows whether it worked, whether the policy is a rules engine, a model, or an agent. The agentic case is the hardest version of the same component.
- System
Surfaces and Assignment
Which parts of the product the system covers, how a visitor is assigned to an arm before they have an account, and how that assignment survives the moment they get one. The component where contaminated results start.
- System
Units Above the Visitor
When the visitor is the wrong thing to randomize: teams that must see one price, marketplaces with two sides, referral loops, and shared devices. The component that decides how much sample a clean comparison costs.
- System
The Metric Ladder and Guardrails
One metric that decides, a short list that can veto, and a record of how each proxy earned its place. The component that stops a fast loop from optimizing the top of the funnel at the expense of the bottom.
- System
Test Design and Traffic
Which test may launch now, on whom, alongside what, at what allocation, for how long, and under what ramp and kill rule. The component that turns a stream of individual tests into a portfolio that can be run at once without the tests reading each other.
- System
Sequential Readouts
When a test is allowed to stop, what it reports when it does, and the checks that run before anyone reads a lift. The component that makes a fast loop trustworthy, because reading a test daily and stopping when it looks good inflates a five percent false-positive rate to about a quarter.
- System
Shipping Rules and the Registry
The rule that turns a readout into a decision, applied the same way to every test, and the registry that records what was asked, what was found, and what was done. The component that makes the tenth quarter cheaper than the first.
- System
The Long-Run Holdback
A small standing share of visitors who never see any shipped change, so that once a quarter the sum of the program's wins can be checked against what the program added. The component that keeps a fast loop pointed at the outcome.
- System
Media Taxonomy and Spend Spine
One vocabulary for channels and tactics, one spend series reconciled to what the business paid, joined to the outcome the budget is meant to move and what it is worth, at a grain that supports the designs you want to run.
- System
Geo Design and Market Matching
Choosing which markets go dark, for how long, and against what, so that the difference afterwards means something. The sample size is the number of markets, not the number of people in them.
- System
The Test Calendar
Experiments as a standing cadence rather than a request, ordered by how much spend rests on a coefficient nobody has validated. The component that makes calibration possible at all.
- System
The Persistent Geo Holdout
A small, rotating share of markets kept dark on the largest channel, so calibration has a recurring anchor without a negotiation. The standing control the calendar cannot provide, when it is worth running, and the price of having one.
- System
Response Curve Estimation
Adstock, saturation, and priors, plus the harder question underneath them: what in the spend history is doing the identifying, and where does the curve stop being estimated and start being assumed.
- System
Calibration
Folding measured effects into the model as priors, and deciding what happens when the experiment and the model disagree. The step that closes the loop between testing and planning.
- System
Budget Reallocation
Turning a set of curves into a plan somebody will run, under the contracts and minimums that make the unconstrained optimum unreachable, and choosing the moves that generate the next experiment.
- System
The Budget Decision Log
Every reallocation recorded with the curves that justified it, the effect it predicted, and what happened. The component that lets a quarter score the quarter before it, and the reason the loop closes instead of restarting.