Marketplace

Choosing Between User, Geo, and Cluster Holdouts

The design decision is about what you can withhold cleanly, not about statistical efficiency. A procedure for picking the unit, in which power is the last question rather than the first.

The Diagnostics That Earn Belief

Balance, overlap, pre-trends, placebo tests, and sensitivity analysis. Which to run, in what order, and the thing they all share: they can falsify a design and never confirm one.

When the Experiment Leaks

Noncompliance, contamination, and spillover are three different leaks that bias in different directions. Intention to treat survives two of them, and the third is only fixed by the design.

Designing a Credible Geo Holdout

A paid social incrementality test carried end to end: market pairing, power, pre-trend diagnostics, a difference-in-differences estimate with its interval, incremental CAC, and what the design cannot settle.

Audience Builder and Capacity Planning

Audience definitions as versioned, reusable objects instead of one-off SQL, and the capacity check that decides whether a test was ever going to work.

Units Above the Visitor

When the visitor is the wrong thing to randomize: teams that must see one price, marketplaces with two sides, referral loops, and shared devices. The component that decides how much sample a clean comparison costs.

Sequential Readouts

When a test is allowed to stop, what it reports when it does, and the checks that run before anyone reads a lift. The component that makes a fast loop trustworthy, because reading a test daily and stopping when it looks good inflates a five percent false-positive rate to about a quarter.

Media Taxonomy and Spend Spine

One vocabulary for channels and tactics, one spend series reconciled to what the business paid, joined to the outcome the budget is meant to move and what it is worth, at a grain that supports the designs you want to run.

Geo Design and Market Matching

Choosing which markets go dark, for how long, and against what, so that the difference afterwards means something. The sample size is the number of markets, not the number of people in them.

The Test Calendar

Experiments as a standing cadence rather than a request, ordered by how much spend rests on a coefficient nobody has validated. The component that makes calibration possible at all.

The Persistent Geo Holdout

A small, rotating share of markets kept dark on the largest channel, so calibration has a recurring anchor without a negotiation. The standing control the calendar cannot provide, when it is worth running, and the price of having one.

Response Curve Estimation

Adstock, saturation, and priors, plus the harder question underneath them: what in the spend history is doing the identifying, and where does the curve stop being estimated and start being assumed.

Calibration

Folding measured effects into the model as priors, and deciding what happens when the experiment and the model disagree. The step that closes the loop between testing and planning.

Budget Reallocation

Turning a set of curves into a plan somebody will run, under the contracts and minimums that make the unconstrained optimum unreachable, and choosing the moves that generate the next experiment.

Content Inventory and Template Map

Every URL mapped to the template that renders it, the intent it serves, and whether it is indexed at all. The unit of change and the unit of randomization are not the same thing, and both have to be known.

Crawl and Index State

Shipping a change is not delivering it. Treatment begins when a page is re-crawled and re-indexed, which happens over weeks, in an order that is not random and is correlated with the outcome.

Template Split Design

The only control organic gets is the other half of a template, so the design question is which pages take the change, stratified by the traffic concentration that decides whether it can detect anything.

Counterfactual Traffic

What to do when the change had to ship everywhere and there is no other half: modeling what the pages would have done, and knowing when a core update has broken the comparison.

Paid and Organic Together

Some of an organic gain on a term you also buy is your own paid traffic arriving free, and reporting the two channels separately can count the same click twice. How much is measured, not assumed, and who owns the number.

The Change Roadmap

Which template gets changed next, ranked by traffic at stake rather than by audit severity, and shipped as a split so the change produces evidence instead of consuming it.