Leads Versus Incremental Qualified Pipeline
A B2B lead program that hit every MQL target, measured by a staged rollout across territories. Leads went up by half. Opportunities and pipeline did not detectably move, the leads converted at two thirds the rate, and the design could never have seen the pipeline effect it was asked about.
- The decision
- Is the lead generation program creating qualified pipeline, or leads?
- The method
- Staged rollout across 24 territories in randomized order, difference in differences on log weekly means with territory-level intervals and randomization inference.
STEP 01
The business decision
A paid lead program, content syndication and sponsored placements feeding gated assets, has been running in half the sales territories for a quarter. It delivers leads at a cost per MQL under target, and marketing wants to roll it out to the rest and grow the budget.
Sales is unconvinced. The reps in the territories that have it say the leads are junk. Marketing says the reps are not following up. Both have dashboards.
STEP 02
The tempting metric
MQLs. The program’s territories produced 6,030 marketing-qualified leads in the quarter, every one of them at a cost below the target, and the territory MQL totals rose by half against the quarter before. On that number the program is the best-performing line in the plan.
The CRM adds that the program sourced 5.9% of new pipeline in its territories and influenced 42%.
STEP 03
Why that may be wrong
An MQL is a definition. A lead qualifies when it meets a rule about title, firm size, and an action taken, and a program can be built to produce leads that meet the rule without producing leads that buy. The rule does not know the difference, and the target is set on the rule.
Sourced and influenced pipeline do not help, for the reason the sales-led system works through: both are computed from the CRM’s own records with no comparison in them. Sourced counts opportunities whose first touch was the program. Influenced counts opportunities the program touched at all. Neither asks how many of those opportunities would have appeared without it, and in a territory where reps are working every account anyway, that is most of them.
The question that decides the rollout is whether territories with the program create more qualified pipeline than territories without it. That needs territories without it.
STEP 04
Data available and missing
Available. Weekly MQLs, opportunities created, and pipeline value by territory, from the CRM, with eight weeks of history before the program started anywhere. The rollout order, which was drawn at random when the program was approved, because the sales leadership would not accept a permanent holdout and would accept everyone getting it eventually.
Missing. Any account-level exclusion. Every account in a treated territory was eligible, so there is no do-not-touch list to compare against, and the unit of comparison is the territory. Also missing: closed-won revenue inside the quarter, since deals take longer than that to close, so pipeline created is the furthest outcome the window can reach.
STEP 05
The target quantity
The effect of running the program on qualified pipeline created per territory, over the first quarter it runs, for territories like these.
The horizon matters more than usual. Leads arrive in the week they are generated. Opportunities arrive over the following eight weeks as reps work the leads. Pipeline arrives as opportunities are sized, which is later still. A readout at any one horizon is an answer about that horizon, and the three will disagree, which is the finding rather than a problem with it.
STEP 06
The strongest feasible design
24 territories, 12 given the program in wave one and 12 waiting a quarter for wave two, with the order drawn at random.
Nobody is held out. Everybody gets the program; only the order was randomized. During the quarter before wave two, the waiting territories are the comparison, and the design is a two-arm test with a hard stop on the day wave two launches. That is the version of a holdout a sales organization will approve, and the designs when randomization is refused component is the argument for it.
The territories differ in size by a factor of 11.1, so the 24 are worth about 17 equal ones. The comparison is on each territory’s own change from its pre-period, which removes the size difference from the level and leaves it in the noise.
STEP 07
Assumptions and failure modes
STEP 08
Worked example and diagnostics
What each horizon showed
Show the data
| Estimated lift | Point | 95% CI |
|---|---|---|
| Leads, 8 weeks | +54.2% | +41.3% to +68.2% |
| Leads, quarter | +55.1% | +43.4% to +67.8% |
| Opportunities, 8 weeks | -6.1% | -22.0% to +13.0% |
| Opportunities, quarter | -0.5% | -15.9% to +17.8% |
| Pipeline, 8 weeks | -7.8% | -34.0% to +28.9% |
| Pipeline, quarter | -4.4% | -29.5% to +29.6% |
Opportunities, week by week
- Wave one (program on)
- Wave two (waiting)
Show the data
| Period | Wave one (program on) | Wave two (waiting) |
|---|---|---|
| W-8 | 104.5 | 87.3 |
| W-7 | 99.4 | 92.1 |
| W-6 | 100.2 | 96.8 |
| W-5 | 102.8 | 109.5 |
| W-4 | 89.2 | 114.3 |
| W-3 | 96.0 | 114.3 |
| W-2 | 103.6 | 95.2 |
| W-1 | 104.5 | 90.5 |
| W1 | 101.9 | 82.5 |
| W2 | 92.6 | 92.1 |
| W3 | 107.0 | 141.3 |
| W4 | 101.9 | 101.6 |
| W5 | 96.0 | 112.7 |
| W6 | 95.1 | 100.0 |
| W7 | 118.0 | 114.3 |
| W8 | 87.5 | 109.5 |
| W9 | 128.2 | 106.3 |
| W10 | 118.9 | 107.9 |
| W11 | 96.8 | 92.1 |
| W12 | 113.8 | 87.3 |
| W13 | 120.6 | 109.5 |
The finding the design could see
The program’s leads convert worse. Across the quarter, treated territories turned 9.0% of their leads into opportunities against 12.7% in the waiting territories, on 18,056 and 6,761 leads respectively, so the intervals do not overlap. Half again as many leads, converting at two thirds the rate, is arithmetic that lands on roughly the same number of opportunities, which is what the opportunity readout shows within its noise.
| Figure | Value | What it contains |
|---|---|---|
| Program MQLs | 6,030 | Leads the program generated. Real, and the only figure here with a comparison behind it that agrees. |
| Sourced pipeline | 5.9% | Opportunities whose first touch was a program lead. A routing fact. Says nothing about whether those accounts would have opened anyway. |
| Influenced pipeline | 42% | Opportunities the program touched at all. Coverage. Rises with volume whether or not the touches did anything. |
| Measured effect on pipeline | −4.4% | Against the waiting territories, with an interval from −29.5% to +29.6%. The only row that asked the question. |
Three of the four are correct about what they count. None of the first three is an answer to whether the program created pipeline, and the fourth is an honest one: not readable at this size.
STEP 09
The estimate, with uncertainty
Effect of the program on qualified pipeline created, treated territories, one quarter
The interval spans a third of the pipeline in either direction. It is not evidence the program did nothing; it is evidence that this design cannot tell, and it says so rather than reporting the point.
- Leads, quarter
- +55.1%
- Lead-to-opportunity rate
- 9.0% vs 12.7%
- Detectable on pipeline
- 51%
The randomization-inference p-value for the pipeline effect, from re-drawing which twelve territories went first, is 0.76. For leads it is below one in a thousand. The design had no trouble seeing the thing the program obviously did and no ability to see the thing it was paid to do.
STEP 10
The recommendation
Do not scale the program on its lead numbers. Hold the budget through wave two, and read pipeline on both waves together at two quarters.
The reasoning:
- The program generates leads. That is established, and it is not in dispute.
- Those leads convert to opportunities at two thirds the rate of the leads the territories were already getting. That is also established, and it is the reason sales and marketing have been arguing: both were right about the thing they could see.
- Whether the program adds pipeline is not established in either direction, and the reason is arithmetic rather than execution. Wave two doubles the treated territories and a second quarter doubles the window, and the pooled read is the first one that could plausibly resolve it.
- Cost per MQL should come off the program’s scorecard until then, because it is the metric the program is built to win and the one that says least.
STEP 11
What this does not establish
- That the program does not create pipeline. The interval includes effects that would justify the budget several times over. It also includes zero and a loss.
- That the leads are bad. They convert worse than the base, which is what colder demand looks like, and a cheaper lead that converts at two thirds the rate can still be the right purchase. That is a cost question this design does not settle.
- That the effect on closed revenue matches the effect on pipeline. Deals close after the window, and the program’s smaller opportunities may close at a different rate.
- Anything about a real company. The territories, the rates, and the CRM figures are simulated so the horizons would disagree the way they do in practice.
STEP 12
What to test next
- Read both waves at two quarters. The design is already running; wave two is the replication. Pooling doubles the units and the window, which is the only route to a pipeline readout at this scale.
- Read at the account grain with the existing-pipeline flag. Accounts already in a deal cycle inflate every figure here. Excluding them, per the account graph, sharpens the comparison and changes the estimand to the one the program was bought for: new pipeline.
- Price the leads, not the pipeline. A lead that converts at 9% against a base of 13% has a value, and the budget decision can be made on that value with an interval while the pipeline question waits.
STEP 13
Technical appendix
The estimator and the intervals Code
For each territory, the log of mean weekly outcome in the read window minus the log of its mean in the eight-week pre-period. The effect is the mean of that change in wave-one territories minus the mean in waiting territories, so each territory counts once regardless of size, and the interval comes from the territory-level spread on t with 22 degrees of freedom. The randomization-inference p-value re-draws which twelve territories went first, 20,000 times, and asks how often a difference at least this large appears under the sharp null of no effect anywhere.
Pipeline is the sum of opportunity values, and opportunity values are lognormal with a wide spread, so a territory’s quarter is dominated by its few largest deals and the pipeline interval is roughly twice the width of the opportunity interval on the same design. That is the ordinary state of B2B measurement, and it is why the recommendation prices leads rather than waiting for pipeline to resolve.
Regenerated from seed 20260917 by the script named
under each figure.
How well did the estimator do? Technical
The simulation planted a lead lift of 50%, an opportunity lift of 8.0%, and a pipeline lift of 6.4%, with program leads converting at 2% against a base of 12.5%. The lead estimate recovered the truth comfortably. The opportunity and pipeline intervals both contain the truth and both contain zero, which is the correct behavior of a design that was never powered to separate them.
That is the lesson. A real program adding six percent to pipeline through 24 territories in a quarter produces a readout that looks like this one, and a team that reads the point estimate concludes the program failed while a team that reads the MQLs concludes it succeeded. Neither is what the data says. The data says: leads yes, conversion rate down, pipeline unknown at this size, and here is what would resolve it.
Context
Best suited for
Sales-assisted programs where a lead source can be turned on by territory or segment in a chosen order, the CRM has territory-level stage history, and the argument between sales and marketing is currently conducted with sourced and influenced pipeline.
Data requirements
- Weekly MQLs, opportunities created, and pipeline value by territory, with at least eight weeks before the program started anywhere
- The rollout order, drawn rather than chosen, and recorded before wave one launched
- Lead routing logs, to confirm program leads did not land in waiting territories
- Stage definitions frozen for the window, since a moving bar is indistinguishable from an effect
What changes by context
- B2B SaaS
- The default case. Opportunity volumes per territory are small enough that the pipeline question needs two waves and two quarters, and the honest early readout is the conversion rate, which is powered from the first month.
- Sales-led
- Reps notice which territories have the program and may work its leads differently, which is a mediator rather than a confounder. Randomizing the territory rather than the account is what keeps the rep’s response inside the effect rather than adjusting it away.
- Marketing-led
- Where the lead volume is high enough that opportunities arrive by the hundred, the same design reads pipeline in a quarter and the two-wave pooling is a refinement rather than a necessity.
When it breaks down
- Fewer than about twenty territories, where the detectable effect on anything past leads exceeds any effect worth finding and the design should not be run for that purpose.
- Programs rolled out everywhere at once, where nothing was assigned and the comparison has to be observational with its assumptions stated.
- Accounts that span territories or leads routed centrally, where the waiting arm receives the program’s output and the estimate is biased toward zero.