Test now or wait until the season ends: A product test that runs into peak trading cannot be separated from the season. Either finish before it starts, or accept a result you will not be able to use.
Read the full article below for detailed insights and actionable strategies.
Attribution by the numbers
iOS tracking loss
Google Brand cannibalization
Klaviyo overstatement
TikTok attribution lag
Seasonality does not add noise to a test. It dominates it. Which makes the calendar a design constraint, not a detail.
Why a season is different from noise
Random variation averages out with more data. A season is a systematic shift affecting every variant at once, in the same direction, at the same time.
If your test window includes the run-up to a peak, the lift you observe includes the season. You cannot subtract it, because you have no version of the same window without it.
The three options, honestly
| Option | What it costs |
|---|---|
| Finish before the season starts | A shorter window, so a wider interval |
| Run through it with a concurrent control | Nothing, if a true control exists |
| Run through it without a control | The result, which will be uninterpretable |
The middle option is the right answer whenever it is available. A control group experiencing the same season removes it from the comparison, which is precisely what a holdout is for.
When there is no clean control
Some tests cannot have one. A sitewide change affects everyone. In those cases the calendar is the only lever, and the choice is to finish early or to wait.
Waiting is usually right and almost never chosen, because a test already built feels wasteful to postpone. The waste is running it into a window where the answer cannot be read.
The other seasonal trap
Launching into a peak and concluding the product is strong. Peak demand lifts almost everything, and a product that merely kept pace with the season has demonstrated nothing about itself.
Judge peak launches against the rest of the catalogue over the same window, never against their own pre-peak baseline.
The read
A causal read on a Google Analytics export returns an estimate with a confidence interval and a coverage share, plus a label where the window cannot support a claim. 99 euro once, refunded if it does not move a budget decision. The interactive demo shows it with no signup.
Related answers
Key Terms in This Article
Analytics
Analytics is the systematic computational analysis of data. It reveals customer behavior and measures campaign performance.
Attribution
Attribution identifies user actions that contribute to a desired outcome and assigns value to each. It reveals which marketing touchpoints drive conversions.
Causality
Causality is the relationship where one event directly causes another, essential for identifying specific actions that drive desired outcomes in marketing.
Confidence Interval
Confidence Interval is a statistical range of values that likely contains the true value of a metric. In marketing analytics, it quantifies uncertainty around estimates, indicating the precision of an outcome or causal effect.
Control Group
Control Group is a segment of an audience intentionally not exposed to a marketing campaign, used to measure the campaign's true causal impact.
Google Analytics
Google Analytics is a web analytics service that tracks and reports website traffic.
Incrementality
Incrementality measures the true causal impact of a marketing campaign. It quantifies the additional conversions or revenue directly from that activity.
Related Articles
Sixty-second versions of these ideas: Causality Engine on YouTube Shorts.
Ready to see your real numbers?
Own the budget? Upload your GA4 export and see which channels drive incremental sales, with confidence intervals, in minutes. Have to defend it? Start with the live demo and take the read to your CFO.
Full refund if you don't see value.
Stay ahead of the attribution curve
Weekly insights on marketing attribution, incrementality testing, and data-driven growth. Written for the person who owns the budget and the person who has to defend it.
No spam. Unsubscribe anytime. We respect your data.
Frequently Asked Questions
Can I run a product test through peak season?
Only with a concurrent control experiencing the same season. Without one, the lift you observe includes the season and cannot be separated from it, because there is no version of that window without the season.
Why is seasonality worse than ordinary noise?
Noise averages out with more data. A season shifts every variant at once in the same direction, so more data does not remove it.
How should I judge a product launched into peak?
Against the rest of the catalogue over the same window, not against its own pre-peak baseline. Peak lifts almost everything, so keeping pace with the season proves nothing about the product.