The Prioritization Log

The scoring model was retrained in January. Its top-ranked accounts progressed at twice the rate of the rest, which went into the quarterly review as proof the model works.

The reps worked the top-ranked accounts. Nobody worked the rest. The model was scored on a comparison between accounts that got attention and accounts that did not, and it will produce that result whatever it ranks.

The decision it encodes

Which accounts did we intend to work, which did we work, and at what odds?

Rep capacity decides how many accounts get worked and by what rule. This component records what that rule did, in a form the next rule can be scored against. The lifecycle system does this with its decision log, and it is the reason a candidate policy there can be evaluated on last quarter’s traffic before it ships. Nothing in a CRM does it by default.

A ranking that only ever sees its own picks

A prioritization model is retrained on outcomes. The outcomes it sees come from accounts that were worked, and the accounts that were worked are the ones it ranked highly last time. Each quarter it grows more confident about a shrinking set of accounts and learns nothing about the ones it never chose, including the ones it was wrong about.

Two things break that. The first is recording, for every account in every period, the probability the rule gave it of being assigned work, so that a comparison between worked and unworked accounts can be reweighted to what it would have been under a different rule. The second is making sure that probability is neither zero nor one anywhere a candidate rule might disagree with the current one, which is the part most exploration schemes get wrong.

What the log carries per account, per period

Field	What it decides
Score and rank	What the model believed, with its version. So a change in outcomes can be tied to a change in the model rather than to the market.
Intended action and service level	Work at a defined level, or baseline, as the rule decided before anyone touched the account. The assignment, in the experimental sense.
Assignment probability	The probability the rule gave this account of being assigned work: one well above the cutoff, zero well below it, and strictly between the two inside the exploration band. The field that makes reweighting possible, and it is the probability of assignment under the rule, not of a rep actually working the account.
Actual activity, capacity, and overrides	Touches, meetings, and hours from the CRM, the capacity available that period, and the reason for any override. Reps deviate from lists, and the deviation has to be visible rather than assumed away.
Stage at assignment	Where the account was when the decision was made, from the account graph. Accounts already in a deal cycle flatter every rule that picks them.
Outcome at horizons	Stage progression at eight weeks, a quarter, and two, from the readouts component, so accrual and reversal are both visible.

Two estimates, and which one the rule is judged on

Reps do not follow lists exactly. Some accounts marked for work are never touched; some marked to skip get worked because a rep knew someone there. The intended action is what the rule assigned, so the comparison that judges the rule is between accounts intended for work and accounts intended for skipping, whatever the reps then did. That is intention to treat, and it is the effect of having the rule.

The effect of the work itself is a second question with more assumptions. The example above has deviation in both directions: assigned accounts that were not worked and skipped accounts that a rep worked anyway. Dividing the assignment effect by the touched share is a Wald estimate, and with two-sided deviation it identifies the effect among accounts whose work the assignment changed only under exclusion, monotonicity, and a first stage strong enough to divide by, none of which the log guarantees. The primary outputs are the effect of the assignment rule, the assignment-to-work first stage, and actual activity by arm. A complier effect is reported in the technical appendix when those assumptions can be defended, and labeled with them, which is the same discipline exposure and decision logging applies to messages that were sent and never seen.

What the log makes possible

A candidate ranking can be scored on the previous period’s log before any rep sees it. Every account has a recorded assignment probability, so the outcomes of accounts the new rule would treat differently can be reweighted by how likely each action was under the old one. The estimate is noisy, it is trustworthy only where the old rule gave both actions some probability, which is the exploration band and nowhere else, and it is available in an afternoon rather than a quarter. Rank candidates by that estimate over the band, and the one worth a live test is usually clear.

The same log is also the first record most programs have of what a rep hour buys near the line. The exploration band produces outcomes on accounts just above the cutoff left at baseline and just below it worked, which is the only evidence about where the cutoff should sit, and the shadow price the capacity component computes is only as good as that evidence.

Maturity, honestly staged

Stage 1: the ranked list stored each period, with the model version. Most programs overwrite it. Keeping it is the difference between knowing the model changed and guessing.

Stage 2: intended action and actual activity, side by side. The gap between them becomes visible, and the readout can report the effect of the rule separately from the effect of the work.

Stage 3: a two-sided exploration band with its probabilities recorded. The scoring gets outcomes under both actions near the cutoff for the first time, and a candidate ranking can be evaluated on the log over the band before it is trusted with capacity.

Stage 4: the readout scores the ranking automatically. Each period’s intended-action effect and each candidate’s off-policy estimate land on a schedule, and the ranking that ships is the one the log supported.

Reweighting the log, and what it cannot do Technical

Off-policy evaluation reweights each logged outcome by the ratio of the candidate rule’s probability of the action taken to the logging rule’s. It is unbiased for the candidate’s value only where the logging rule gave every action the candidate might take a nonzero probability. A rule that always works its top accounts gives “not worked” zero probability there, so a candidate that would drop one of them has no support and cannot be evaluated, however much exploration happened below the cutoff. That is why the band is two-sided, and why the evaluation is restricted to it: outside the band the two rules agree and there is nothing to estimate. Small probabilities inside the band make the weights large and the estimate noisy, which is the price of a narrow band.

With a few hundred worked accounts a period and a band of a tenth of that, the estimates are wide. At 2,000 accounts, 400 worked, and a band covering 10% of worked capacity on each side of the cutoff, the band holds about 80 accounts a period, and a progression rate of a few percent gives a handful of events to learn from. The log accumulates, and over a year it is the only evidence about the cutoff that exists. It does not substitute for a staged rollout when one is available.

The probabilities have to be the ones the rule used, not reconstructed afterwards from who got worked. A propensity estimated from the outcome data is a model of the reps’ behavior, and reweighting by it recovers the reps’ preferences rather than the rule’s effect.

Context

Best suited for

Programs where a scoring model produces the list reps work from, capacity is binding, and the model is retrained on the outcomes of its own recommendations.

Data requirements

• The ranked list each period with the model version, stored rather than overwritten
• Intended action per account as the rule decided it, before any rep touched the list
• Activity per account from the CRM, at a grain that distinguishes a touch from a meeting
• Stage history from the account graph, so outcomes can be read at more than one horizon

What changes by context

B2B SaaS

The natural home, since a scored list already exists in most programs and capacity is already binding. The exploration slice is the new ask, and it is smaller than it sounds against a list the reps already ignore parts of.

Sales-led

Reps will recognize exploration accounts if the tooling marks them, and the assumption above is the one most likely to fail. The slice has to look like the rest of the list.

Partner-led

The unit is the partner rather than the account, the list is shorter, and a tenth of it may be two partners. The log still works; the exploration slice mostly does not, and the design has to rely on staged timing instead.

When it breaks down

• Lists short enough that a tenth of capacity is a handful of accounts, where exploration produces no usable outcomes and the log is a record rather than an instrument.
• Teams where reps ignore the list entirely, in which case the intended action assigns nothing and the log records the reps’ preferences under the name of a rule.
• Programs whose model is retrained on the log without the exploration slice, which reproduces the feedback loop this component exists to break.