How do I test if email is incremental?
Hold out a random slice of the people you email from one flow or campaign. Send to everyone else as usual, and compare how many in each group bought. The gap is what email caused. The revenue your email platform reports is credit, not a test.
By Joris van Huët, Founder & CEOUpdated 7 min read
Run the numbers for your store: the free email marketing ROI calculator.
Usually with a holdout. Pick a random slice of your list and keep it out of one flow or campaign, while everyone else gets emails as normal. After a few weeks, compare how many people in each group bought. The gap is what email caused. The revenue figure in your email platform is credit, not that gap.
The usual answer is to read the revenue your email platform puts next to each flow, or the Email row in GA4. Both are reports of credit. Neither asks what would have happened if the email had never gone out.
Klaviyo's attribution guide says it uses a last touch model by default, with 5-day windows for email clicks and for email opens. So an order placed four days after an open counts for the email, whatever else the buyer saw. GA4 has the opposite habit: it can only file a visit under Email when the link says so.
What one store's data shows
One store's anonymised GA4 export, 1 January 2024 to 21 August 2026. It holds shares of revenue only: no ad spend, no order counts.
| What the export shows | Share of revenue | Source cell |
|---|---|---|
| Direct, in last click, first click and touched views | 57.7% | Channels sheet, Direct row |
| Touched view, every channel added up | 110.4% | Channels sheet, Touched column total |
| Journeys with 1 touch (0.5 days to buy) | 79.5% | Journeys sheet, 1 touch row |
| Journeys with 2 to 3 touches (12.5 days to buy) | 12.2% | Journeys sheet, 2-3 touches row |
Start with Direct. It holds 57.7% of revenue in all three views of that store's Channels sheet. Google's help page on (direct) / (none) traffic says source information is lost when links lack UTM parameters. It even names email campaigns as a place to check. So if your email links go out untagged, part of what email did can hide in Direct.
The touched view in that store's Channels sheet adds up to 110.4%, by design. A journey that touched two channels counts in both. Email platforms keep score from their own side in the same spirit. Add Klaviyo's email credit to what Meta and Google Ads claim, and the total can pass what you sold.
Then speed. Journeys with 1 touch hold 79.5% of revenue in that store's Journeys sheet, and took 0.5 days to buy. If your export looks like that, many buyers arrive with their minds made up. An email that lands the morning they planned to buy still collects the credit, because it was the last thing they opened.
The slow journeys cut the other way. In that store's Journeys sheet, journeys with 2 to 3 touches took 12.5 days to buy, well past a 5-day window. A send that starts a two-week decision can miss its own credit.
None of these rows can say whether email caused a single sale. The export holds no spend, no order counts and no test. It does tell you how to set one up: tag every email link, and keep counting well past five days.
Why does the email report disagree with GA4?
Because each one counts something different, and neither counts what the send changed.
Your email platform counts touches. Klaviyo's default is last touch, with 5-day windows for email clicks and for email opens. Its own settings guide warns that a longer open window may introduce a bias towards opens. Bot clicks are left out by default on new accounts. Apple Mail Privacy Protection opens are a separate switch, under Settings, then Attribution.
GA4 counts visits it can label. By GA4's default channel rules, a session lands in Email only when its source or medium matches email. An untagged click can fall into (direct) / (none) instead. And Klaviyo changed its default utm_medium to the message type, such as email, on 2 June 2025. Accounts that saved their UTM settings before then still send the old values, campaign and flow.
Neither one holds anybody back. Klaviyo's own global holdout card makes the switch you need. It reports total conversions made by the profiles in each group, not attributed ones. That is the number a test runs on.
So the email report leans high, and GA4 can lean low. Only a holdout compares people who got the email with people who didn't.
What can an email holdout not tell you?
Which email did the work. A holdout compares sending with not sending. Klaviyo draws the same line: a holdout group gets nothing, while an A/B test compares two versions. Subject lines and offers need the A/B test.
Email on its own, if you use the global holdout. Klaviyo's global holdout groups skip campaigns and flows on every channel those messages use. So they measure email, SMS and push together. To test email alone, split one email-only flow instead.
The long run. A few weeks can't show list fatigue, unsubscribes or the slow pull of a brand people hear from every week. Klaviyo recommends running global holdout groups for 3 months to gather enough data.
Much of anything, on a small list. Klaviyo's global holdout groups need at least 400,000 profiles. Below that, test one busy flow at a time, and treat a small gap between the groups as possible luck.
What to do this week
- Check that email shows up in GA4 at all. In GA4, open Reports, then Acquisition, then Traffic acquisition, and find the Email row. Pass: Email has sessions on the days you sent. Fail: there is no Email row, or a tiny one, so links are untagged or still carry the old utm_medium values.
- See how much of your email credit rides on opens. In Klaviyo, open the account menu, then Settings, then Attribution, and click Compare model. Preview your flows with email opens unticked, then leave without applying. Pass: flow revenue barely moves. Fail: it drops sharply, so much of the credit rests on opens, not clicks.
- Choose the flow you will hold out. In Klaviyo, open Flows and pick the one with the most people entering each week. Pass: you can name the flow, its weekly entrants and a start date. Fail: no flow has steady entrants, so plan a longer test on the busiest one, then follow the step-by-step version.
Check the homework. Your GA4 Attribution paths export already holds the evidence. Causality Engine reads that one file and shows what each channel caused next to what last-click gave it, in 1 to 2 minutes, for €99 once (excluding VAT), refundable within 30 days. Check the homework
Sources, 1 October 2026: Understanding Klaviyo message attribution (Klaviyo Help Center); How to change your attribution model (Klaviyo Help Center); Getting started with global holdout groups (Klaviyo Help Center); Understanding UTM tracking in Klaviyo (Klaviyo Help Center); How to A/B test flow branches (Klaviyo Help Center); Understand (direct) / (none) traffic (Google Analytics Help); Default channel group (Google Analytics Help); Traffic acquisition report (Google Analytics Help)
Related answers
Frequently asked questions
How big should an email holdout group be?
Big enough that each group gets a fair number of buyers, and small enough that holding email back costs little. Klaviyo recommends about 5% of marketable profiles for its global holdout groups. On a smaller list, hold out a larger slice of one flow, or run the test longer.Should I test email flows or campaigns first?
Flows, usually. A flow starts on a trigger, so a random split right after it sorts each new person into a group before any email goes out. Campaigns go to the whole list at once, so testing them needs a group kept out of every send.Can I test email incrementality with a small list?
Yes, one flow at a time. Klaviyo's global holdout groups need at least 400,000 profiles, but its guide to splitting a flow at random sets no such minimum. Expect a longer test, because few buyers per group make a small gap easy to mistake for a real one.
Go deeper: Causal attribution, explained.
Sixty-second versions of these ideas: Causality Engine on YouTube Shorts.
Keep reading
Terms in this article
- AttributionAttribution identifies user actions that contribute to a desired outcome and assigns value to each. It reveals which marketing touchpoints drive conversions.
- Attribution ModelAn Attribution Model defines how credit for conversions is assigned to marketing touchpoints. It dictates how marketing channels receive credit for sales.
- ConversionConversion is a specific, desired action a user takes in response to a marketing message, such as a purchase or a sign-up.
- Google AnalyticsGoogle Analytics is a web analytics service that tracks and reports website traffic.
- Holdout TestA holdout test is an experiment where a portion of the audience does not see a campaign. This measures the campaign's true incremental impact.
- IncrementalityIncrementality measures the true causal impact of a marketing campaign. It quantifies the additional conversions or revenue directly from that activity.
- Incrementality TestingIncrementality Testing measures the additional impact of a marketing campaign. It compares exposed and control groups to determine causal effect.
- UTM ParametersUTM Parameters are URL tags marketers use to track campaign effectiveness across traffic sources. They provide data for accurate campaign tracking and attribution in analytics platforms.