Skip to content
Was It Causal?

Systems · Calculator

Geo holdout designer

Most geo tests that 'find nothing' were never powered to find the effect anyone cared about. This estimates what lift your design can detect, while you can still change the design.

Part of Systems. The audience-level equivalent lives in audience builder and capacity planning.

Your design

Split evenly between treated and held out.

Standard deviation of log weekly conversions within a market. Around 0.18 is typical; noisy categories run 0.30 or higher.

The smallest effect that would change a decision. Not the effect you hope for.

%

Two-sided α = 0.05, power = 80%.

Verdict

With 40 markets and typical variability, a 5% lift needs roughly 4 weeks to be detectable.

Run the numbers for your own design on the left.

Minimum detectable effect by duration

The smallest lift you could reliably detect, by how long you run.

Duration Detectable lift Enough?
2 weeks 11.9% Enter your numbers
3 weeks 9.6% Enter your numbers
4 weeks 8.3% Enter your numbers
6 weeks 6.7% Enter your numbers
8 weeks 5.8% Enter your numbers
10 weeks 5.2% Enter your numbers
12 weeks 4.7% Enter your numbers
The formula Technical

For n markets per arm, within-market log-scale standard deviation σ, and W weeks: 

SE  = σ · √(2/n) / √W
MDE = exp((z1−α/2 + zpower) · SE) − 1

The estimate is on the log scale, so the result is a multiplicative lift. This is the same calculation used in the geo holdout worked example, where σ is measured from the pre-period rather than assumed.