How Long Should an Incrementality Test Run? A Sample-Size Guide for Shopify Brands

How Long Should an Incrementality Test Run?

Run your test until each arm has accumulated roughly 16 ÷ (expected relative lift)² orders. At a 50/50 split, divide by half your weekly order volume to get the number of weeks. Below that threshold the test cannot distinguish a real effect from noise.

That is the whole answer. The rest of this guide explains where the number comes from, gives you a duration table by order volume, and, more importantly, tells you what to do when the arithmetic says your brand is too small to run the test at all. That last part is where most Shopify brands actually live, and almost nobody writes about it honestly.

Why Most Incrementality Tests Are Dead on Arrival

Incrementality testing is the gold standard for measurement, and for good reason: it is the only method that directly observes what happens when you turn spend off. Every other approach infers it. So the advice "just run a holdout" has become the reflexive answer to any attribution question.

The problem is that a holdout test is a statistical instrument with a resolution limit, and nobody checks the limit before pointing it at their business.

This is not a subtle statistical footnote. It is the single most common way incrementality testing goes wrong, and it produces confidently wrong budget decisions in both directions: brands cutting channels that worked, and brands scaling channels that did not.

The three numbers that determine whether your test can work

Before you design anything, you need three inputs:

1. Baseline conversion rate or order volume. How many orders happen in a given period without any change.
2. Minimum detectable effect (MDE). The smallest lift you would actually change your budget over. Not the lift you hope for: the lift that would change a decision.
3. Tolerance for error.

Almost everyone skips step 2.

The 16-Over-Lift-Squared Rule

Here is the piece of arithmetic worth memorising.

Orders per arm ≈ 16 ÷ (relative lift)²

The elegance is that it does not depend on your conversion rate.

Halving the effect you want to detect quadruples the data you need.

Duration Table: Weeks to Significance by Order Volume

Convert orders into calendar time with a 50/50 split:

Weeks ≈ 32 ÷ (relative lift² × weekly orders)

* Never run shorter than two full weeks regardless of what the arithmetic says. You need to cover complete weekly cycles, or day-of-week composition differences between arms will contaminate the result.

Anything above roughly eight weeks should be treated as impractical, not merely slow. Over a long window your creative changes, competitors move, seasonality shifts, and the assumption that the only difference between arms is the treatment quietly stops being true.

The Test Feasibility Ladder

Rather than asking "can I run an incrementality test," ask which rung of this ladder your brand is standing on. Each rung has a different correct answer.

The detectable lift in eight weeks follows from the duration formula above: about 2 ÷ √(weekly orders), so 20% at 100 orders a week, 10% at 400 and 5% at 1,500.

If you are doing 300 orders a week and an agency proposes a two-week creative-level incrementality test, the proposal is not aggressive: it is arithmetically void.

A Worked Example: The €40,000 Question

Illustrative example. Figures are constructed to demonstrate the method, not drawn from a specific customer.

A Shopify skincare brand spends €40,000 a month on Meta and does 900 orders a month (about 210 a week) at €95 average order value. The founder wants to know whether Meta is worth it.

The causal read they attempt. They run a two-week 50/50 geo holdout. Result: treatment cities ordered 6% more than control. They conclude Meta drives 6% incremental revenue, roughly €5,100 a month against €40,000 of spend, and prepare to cut Meta entirely.

Why that conclusion is unsupported. Two weeks at 210 orders/week is 420 total orders, 210 per arm. Consult the rule: 210 orders per arm supports an MDE of √(16 ÷ 210) ≈ 27.6%. The test did not show Meta is weak. It showed nothing at all.

Common Mistakes

• Designing without an MDE. Decide the lift threshold that would change your budget before you start.
• Peeking and stopping early. Checking results daily and stopping when p < 0.05 inflates your false positive rate dramatically. Fix the duration in advance and honour it. Log the pre-registration in a marketing experiment tracker.
• Unbalanced splits for "safety." A 90/10 holdout feels safer but is far weaker than 50/50. Effective sample size is governed by the smaller arm; a 10% holdout needs roughly 2.8× the total volume of a balanced test.
• Testing during anomalous periods. A holdout spanning Black Friday measures Black Friday, not your channel.
• Ignoring cross-arm contamination. If your control geography sees your ads through spillover, or your holdout audience still receives branded search and retargeting, your control is not clean and your measured lift is biased toward zero.
• Confusing a null result with a zero. "Not statistically significant" means "this test could not tell." It is not evidence the channel does nothing.
• Forgetting regression to the mean. Testing a channel right after an unusually bad month will flatter the result. Regression to the mean is a persistent contaminant in short tests.

Pre-Flight Checklist

Before launching any incrementality test:

• I have written down the MDE: the lift that would change my decision.
• I have computed required orders per arm: 16 ÷ (MDE)².
• I have divided by half my weekly orders and the answer is 8 weeks or fewer.
• The window contains no promotions, launches, or seasonal peaks.
• Split is as close to 50/50 as the business will tolerate.
• Arms cannot contaminate each other via audience overlap or geographic spillover.
• I have pre-registered the duration and stop rule, and I will not peek.
• I have calculated the revenue cost of the withheld arm and accepted it.
• I know what I will do if the result is null.

If any box is unchecked, you are not ready. If box three fails, you are on the wrong rung of the ladder and should be modelling rather than experimenting.

When You Cannot Run the Test

These are weaker than a clean randomised experiment and stronger than any rules-based attribution model. The attribution maturity model is not a ladder you climb by buying a more expensive tool; it is one you climb by matching method to the evidence your order volume can actually support.

Key Takeaways

• Required orders per arm ≈ 16 ÷ (relative lift)². Memorise it.
• Detection cost scales quadratically: halving the MDE quadruples the data required.
• Never run shorter than two weeks or longer than eight.
• A null result means the test could not tell. It is not evidence of zero effect.
• Below ~400 weekly orders, channel holdouts mostly cannot resolve decision-relevant effects. Model instead of experimenting: that is a legitimate strategy, not a consolation prize.

Incrementality testing earned its reputation honestly. But a method that requires more evidence than your business generates is not rigorous when you use it anyway: it is just expensive guessing with a scientific vocabulary. The discipline is knowing which instrument your data can support, and reaching for causal inference on historical data when the experimental one is out of range.

For €99, upload any historical GA4 period and get causal attribution for every channel in 1–2 minutes: no pixel, no migration.