Designs When Randomization Is Refused

The proposal is a clean account-level holdout. Two thousand target accounts, randomize half into a control arm, run the program against the rest, read pipeline in two quarters.

The VP of Sales says no, and is right to. Those are named accounts with owners and quotas attached, the list exists because somebody believes those companies will buy, and asking a rep to leave four hundred of them alone for six months is asking them to miss a number for a measurement they did not request. The answer will be no this quarter and every quarter after it.

The proposal was the wrong one. Nobody needed the reps to stop selling. The question was whether the program, added on top of the selling, created pipeline, and that question is answered by a control arm that receives the selling and not the program.

The decision it encodes

Where does a comparison come from when nobody will let you leave accounts alone?

This is the constraint that shapes the whole system. Every other system on this site can withhold its unit if it is willing to pay for it. Here the payment for withholding the selling is somebody’s commission, and that refusal is permanent. What is not refused, most of the time, is withholding the marketing treatment while the selling continues, and the design ladder starts there.

The ladder, from what can usually be assigned to what rarely can

The program, at the account, with sales left alone. Eligible accounts are blocked within rep, segment, and stage at assignment and randomized to receive the incremental program or business as usual. Reps sell to both arms exactly as before. The intention-to-treat comparison is the total effect of adding the program under the existing sales policy, including whatever extra rep response it prompted, which is the number the budget decision needs. This is refused far less often than a do-not-touch list, because it asks nobody to stop working an account, and it is the design most programs could have run and did not.

Territories or partners, as clusters. Where the treatment cannot be delivered per account, where accounts in a territory talk to each other, or where the program would redirect a rep’s fixed hours from control accounts to treated ones, the territory is the unit. Reps are already grouped, and a territory can receive a program without any account being singled out. The sample size is the number of territories, which is usually small enough to be uncomfortable and large enough to be worth doing.

Timing, as a randomized rollout. Where everybody must eventually get the program, randomize the order. Half the territories in January and half in April is a stepped-wedge design, and naming it says what has to be handled: calendar effects, exposure duration, clustering, and effects that change with time since adoption, which the staggered rollout lesson shows a naive difference-in-differences misreads. The enthusiastic regions in January and the rest whenever they ask is a selection problem wearing a schedule.

Encouragement, where only a signal can be assigned. When the work itself cannot be withheld but an alert, a recommendation, or an invitation can be randomized, the assignment is the encouragement and the estimate is the effect of the signal, scaled to the accounts whose treatment it changed under the usual complier assumptions.

Segments and tiers, with care. Rolling out to mid-market first and enterprise later is not a comparison, because the timing is entangled with two populations that differ in everything. Randomize the order within a tier, or stratify so that comparable accounts are assigned in both directions, rather than letting one tier stand as the control for another.

Matching, when even that is refused

Sometimes nothing can be assigned, and the response is a matched observational design with its assumptions stated rather than a claim dressed as a test.

Match treated accounts to untreated ones on what predicts the outcome: prior pipeline, size, industry, product usage, and above all prior engagement with sales. Check that the pre-period trajectories track. Then report the estimate as what it is. When randomization is impossible covers the family of designs and what each one buys.

The weakness is specific and worth naming rather than burying. Accounts get selected into a program because somebody believed they would respond, and that belief is usually informed by signals no matching variable captures.

Prior sales effort is a confounder; induced sales effort is a mediator

This is the distinctive problem, and the reason B2B is harder than anything else on this site. It has two halves that are easy to run together.

Effort a rep had already put into an account before the program is a confounder. It predicts which accounts get selected into a program and it predicts the outcome, so a design has to balance it: block on it when assigning, match on it when nothing was assigned.

Effort the program caused is a mediator. Marketing touches an account, a rep notices the engagement and works it harder, a deal appears. Every part of that is real and desirable, and the rep’s effort is the path the program works through. Adjusting for it does not isolate the marketing; it changes the question from the total effect of the program to a direct effect that bypasses the rep, which is rarely what anyone wants and needs assumptions about the mediator that no design here supplies. The estimand this system reports is the total effect of assigning the program under the normal sales response, and rep activity is recorded by arm as the mechanism and as compliance, not adjusted away.

A program can move pipeline without creating it

Rep hours are fixed inside a quarter. When a program produces a strong signal on a treated account, the rep spends four extra hours on it, and those hours came from another account in the same territory. The treated account progresses, the control account it borrowed from does not, and an account-level comparison reads the sum of both as the program’s effect while the territory’s total pipeline may not have moved at all.

That is interference through the shared capacity constraint, and it is where the program loop and the allocation loop touch. Every readout reports both the effect among assigned accounts and the effect on total pipeline across the rep or territory whose hours were redirected. Where the two disagree, the program moved pipeline rather than creating it, and where the redirection is expected to be large, the territory is the unit to assign in the first place.

Maturity, honestly staged

Stage 1: any assigned comparison at all. A staged rollout where the order was drawn rather than chosen. Most programs have never had one.

Stage 2: randomized territories or tiers, with the strata declared. The comparison stops depending on which regions volunteered.

Stage 3: pre-period trajectories checked and a detectable effect stated before launch. A design that cannot see a result worth acting on is identified while changing it is still cheap.

Stage 4: a standing rollout calendar. New programs are staged by default, so evidence accumulates without a separate negotiation every time.

What a dozen territories can detect Technical

The estimator is a difference in differences across territories, and its precision follows the number of territories and the variance of their period-over-period change, not the number of accounts inside them. With n territories split evenly and a standard deviation σ of the quarterly change after matching, the smallest detectable effect is about 2 · 2.8 · σ / √n, which for twelve territories is roughly 1.6σ.

That is a large effect, and territory pipeline is volatile because a single seven-figure deal moves a quarter. Two things follow. Use an outcome less lumpy than closed revenue, such as the count of accounts reaching a defined stage, which has far lower variance than the value of what they reach. And use the pre-period value as a covariate, which helps more here than almost anywhere because territory performance is strongly autocorrelated.

The formula is optimistic with a dozen clusters. Territory outcomes are unequal and autocorrelated, and a normal approximation on eleven degrees of freedom understates the interval. Plan from a placebo simulation on the territories’ own history with the planned pairing and estimator, and read with randomization inference over the territory assignment, which the precision lesson shows can widen a conventional interval by half. Even then, most territory designs are underpowered for anything but a large effect. That belongs in the design document rather than in the readout, because the conclusion of an underpowered test is that nothing was learned, and it is far easier to say that before the result exists.

Context

Best suited for

Programs where accounts are named, owned, and quota-bearing, so an account-level holdout will not be authorized however well it is argued for.

Data requirements

Territory and segment assignments as they stood at launch, not as they stand now
Pipeline and stage-entry counts by territory and quarter, for several quarters before the program
A record of which accounts were already in a deal cycle at assignment
Reorganizations, rep changes, and quota changes as dated events, since those are the shocks that break the design

What changes by context

B2B SaaS: Product usage gives an early outcome that moves in weeks rather than quarters, which is the most useful thing available for shortening a design here. Validate it against closed revenue once before relying on it.
Sales-led: The design has to survive a sales organization that will reassign territories mid-test for reasons unrelated to the program. Fixing the assignment at launch and reporting reassignments as attrition is the only version that holds.
Partner-led: Partners are the natural randomization unit and behave like territories, with the extra problem that a partner can decline to participate, so the assigned and treated sets diverge in a way correlated with partner quality.

When it breaks down

Very few territories, where a single reorganization or one large deal carries the result; there is no universal count, and the placebo simulation on the actual territories says where the line is.
Programs rolled out to everyone at once, where nothing was assigned and the output is a matched comparison with its assumptions stated.
Organizations that reassign territories continuously, so the unit of randomization dissolves during the test.
Effects working entirely through prompting rep effort, where the program and the response cannot be separated and the combined number is the only one available.