From Segment to Audience to Treatment Policy

The winback campaign performed better this quarter than last. Somebody asks what changed and is told the targeting was improved.

That sentence could describe five different changes. The eligibility rule could have moved, so a different population received the campaign. The model could have been retrained, so the same population was ranked differently. The cut-off could have moved, so more or fewer people were contacted. The offer could have changed for the same people. Or the holdout share could have changed, so the comparison itself is different.

Each one makes the two quarters incomparable in a different way, and the word targeting covers all five.

Five objects, and what each one is

The five things “targeting” can mean

Object	What it is	What breaks when it changes
Segment	A description of a kind of customer. No population attached, no version, no eligibility.	Nothing directly. It is language, and it breaks things by being mistaken for an audience.
Audience	A versioned, reproducible population with consent and eligibility resolved.	The population. Results before and after are about different people.
Score	A prediction attached to a person, from a model with a version.	The ranking. Same population, different order, so the treated subset changes.
Assignment	The draw that puts a person in an arm. The only object that creates a comparison.	The comparison itself. Change the holdout share and the readout is not the previous readout.
Policy	The rule mapping scores and constraints to an action, including no action.	What people receive. Same population, same ranking, different treatment.

The segment is the one that causes trouble by being harmless

A segment is a way of talking: high-value churn risks, lapsed browsers, power users. It has no version, no eligibility, no consent resolution, and no reproducible membership.

That is fine until somebody builds a campaign against a segment, because then the population is whatever a query returned on the day, and it will return something different next month for reasons nobody recorded. The transition from segment to audience is the step where a description becomes a reproducible object, and skipping it is the most common reason lifecycle results cannot be compared across quarters.

The score and the policy are not the same decision

A score ranks people. A policy decides what happens to them, and it is where cost, margin, capacity, fatigue, and the option of doing nothing enter.

Conflating them produces a specific and expensive failure: treating a score threshold as a decision. The cut at the top decile is not a decision, it is a convention, and the right cut depends on what a contact costs and what the outcome is worth. Two programs with identical models and different cost structures should treat different fractions of the same ranking, and the break-even rule is what separates them.

What to record, so the next comparison is possible

Four fields, attached to every readout, and none of them require new infrastructure:

The audience version, so the population is identifiable. The model version, so the ranking is. The policy version, including the threshold and the constraints in force. And the assignment salt and share, so the comparison can be reconstructed.

A readout carrying those four can be compared to any other readout carrying them, and the differences can be attributed. A readout carrying none of them is a number about a quarter, which is a different and much weaker thing.

Why the policy is the only one that can be evaluated off-policy Technical

Of the five objects, only the policy maps a state to an action with a probability, and that is what makes off-policy evaluation possible. Given logged decisions with the probability each action carried, a candidate policy can be scored on historical traffic by reweighting, which is the mechanism described in exposure and decision logging for the lifecycle system.

Nothing equivalent exists for the other four. A new audience cannot be evaluated on old data because the population is different, and a new model can only be evaluated on its predictions rather than on its consequences, which is the distinction between predicting who converts and estimating who was moved. That asymmetry is a practical argument for keeping the policy as a separate, explicit object: it is the only layer where a change can be priced before it ships.

It also explains a common confusion. Offline model validation reports how well the score ranks outcomes, and teams read that as evidence the program will improve. It is evidence about the score. Whether the program improves depends on the policy built on it and on the constraints that policy runs under, neither of which the validation touched.

Context

Best suited for

Programs running the same campaigns repeatedly with models and rules that evolve, where somebody is regularly asked why performance moved.

Data requirements

• Versioned audience definitions, dated, with the eligibility rule stored rather than described
• Model version recorded per scored person rather than per model release
• Policy version including thresholds and the constraints in force at the time
• Assignment salt and holdout share, so the comparison can be reconstructed later

What changes by context

Lifecycle-led

The setting this is written for. Dozens of campaigns share audiences and models, so one retrain moves many programs at once and the movements are attributed individually to whatever each team changed that month.

B2C subscription

Outcomes mature over months, so a readout usually spans more than one version of everything. Recording versions per cohort rather than per campaign is what makes a retention readout interpretable.

B2C ecommerce

Promotional calendars change the policy without anybody calling it a policy change, since the offer is the treatment. An offer change is a policy version and should be recorded as one.

When it breaks down

• Single-campaign programs with a static rule, where there is nothing to version and the distinction is overhead.
• Organizations that cannot version audiences because the definition lives in a tool that keeps no history, where period comparisons are not available.
• Fully automated policies that change continuously, where discrete versions do not exist and the comparison has to come from a persistent holdout rather than from period-over-period reporting.

What this does not establish

• That versioning makes a comparison causal. It makes two readouts comparable to each other; whether either one contains a counterfactual depends on the assignment, which is a separate object.
• That a model version change explains a performance movement. It makes it a candidate, and separating it from everything else that changed requires a holdout that spans both versions.
• That these five are the only objects that matter. Creative, timing, and channel are further dimensions, and the argument here is about the ones that get collapsed into the word targeting.
• Any claim about how much of a typical quarter-over-quarter movement is version drift. That is measurable on a specific program with version history and cannot be assumed.