Test now or wait until the season ends
A product test that runs into peak trading cannot be separated from the season. Either finish before it starts, or accept a result you will not be able to use.
Three platforms, three answers, one budget to defend. Your GA4 export already holds the answer, and the first finding is free and never leaves your browser. Get your first finding free
By Joris van Huët, Founder & CEOUpdated 2 min read
Seasonality does not add noise to a test. It dominates it. Which makes the calendar a design constraint, not a detail.
Why a season is different from noise
Random variation averages out with more data. A season is a systematic shift affecting every variant at once, in the same direction, at the same time.
If your test window includes the run-up to a peak, the lift you observe includes the season. You cannot subtract it, because you have no version of the same window without it.
The three options, honestly
| Option | What it costs |
|---|---|
| Finish before the season starts | A shorter window, so a wider interval |
| Run through it with a concurrent control | Nothing, if a true control exists |
| Run through it without a control | The result, which will be uninterpretable |
The middle option is the right answer whenever it is available. A control group experiencing the same season removes it from the comparison, which is precisely what a holdout is for.
When there is no clean control
Some tests cannot have one. A sitewide change affects everyone. In those cases the calendar is the only lever, and the choice is to finish early or to wait.
Waiting is usually right and almost never chosen, because a test already built feels wasteful to postpone. The waste is running it into a window where the answer cannot be read.
The other seasonal trap
Launching into a peak and concluding the product is strong. Peak demand lifts almost everything, and a product that merely kept pace with the season has demonstrated nothing about itself.
Judge peak launches against the rest of the catalogue over the same window, never against their own pre-peak baseline.
The read
99 euro once, refundable within 30 days, no questions asked.
Related answers
Frequently asked questions
Can I run a product test through peak season?
Only with a concurrent control experiencing the same season. Without one, the lift you observe includes the season and cannot be separated from it, because there is no version of that window without the season.Why is seasonality worse than ordinary noise?
Noise averages out with more data. A season shifts every variant at once in the same direction, so more data does not remove it.How should I judge a product launched into peak?
Against the rest of the catalogue over the same window, not against its own pre-peak baseline. Peak lifts almost everything, so keeping pace with the season proves nothing about the product.
Go deeper: Causal attribution, explained.
Sixty-second versions of these ideas: Causality Engine on YouTube Shorts.
Have an audience of ecommerce operators? Earn 20% recurring commission by recommending the €99 read.
Keep reading
Terms in this article
- AttributionAttribution identifies user actions that contribute to a desired outcome and assigns value to each. It reveals which marketing touchpoints drive conversions.
- Control GroupA control group is the part of an experiment that does not get the treatment being tested, so its outcomes stand in for what the treated group would have had without it. In marketing it is the audience, or the set of regions, held back from a campaign.
- IncrementalityIncrementality is the part of an outcome, such as orders or revenue, that a marketing activity caused: the extra results compared with what would have happened without it. It is a causal effect, not the number of conversions credited to a channel.