Is your welcome popup discount incremental?
A welcome popup discount only pays if it creates orders that would not have happened. Check how fast signups turn into orders and how many first orders carry the code, then hold the popup back from some visitors and compare margin per visitor.
By Joris van Huët, Founder & CEOPublished 6 min read
Run the numbers for your store: the free holdout test planner.
A welcome discount pays every new buyer who takes it, including the ones who were already on their way to checkout. Whether your popup earns its cost depends on how many orders it creates, not on how many emails it collects. Your email tool reports the second number.
A funnel teardown doing well on ecommerce YouTube this week walks through a mushroom coffee brand's spin-to-win popup, the SMS ask that follows it, and a welcome flow whose offer does not match the popup (the video). It is a useful look at what a popup does. What nobody can see from the outside is what the popup adds, and that is the only number that decides whether it pays.
Two kinds of people fill in the popup
Everyone who enters an email for a code belongs to one of two groups:
- People who would not have bought today. The code and the emails that follow bring some of them back to buy. That is the popup working.
- People who were going to buy anyway. They see a code, take it, and pay less for the order they were already placing. That is a cost.
Both groups look the same in every report you have, and your email tool cannot tell them apart either. Klaviyo, for example, attributes a placed order to a message only when it falls inside an attribution window after an open or a click, and its default windows for email opens and clicks are five days (Klaviyo Help Center). A buyer who opens the welcome email to copy the code on the way to checkout counts as welcome flow revenue by that rule, whether or not the email changed anything. That is the gap between attribution and incrementality.
How to tell from your own data
Four numbers, all from your own store and email tool:
- Time from signup to first order. Export signups and first orders for the same people. When a large share of them order within minutes of signing up, the popup is mostly a coupon handed out at checkout.
- Code use among first orders. If nearly every first order carries the welcome code, the code has become the price new buyers pay, not an incentive.
- Where people sign up. Signups on product pages and in the cart are much closer to a purchase than signups on the homepage or a blog post.
- First order margin with and without the code. The difference per order, times the number of coded first orders, is what the popup costs you each month before it earns anything.
That last number is the bar. The popup has to create enough extra orders, at full margin, to cover the discount on every order that carries the code.
| Number | Where to find it | A worrying answer |
|---|---|---|
| Signup to first order | Email tool plus store orders | Most first orders arrive within minutes of signup |
| Code use | Store orders, first orders only | Nearly every first order uses it |
| Signup page | Form analytics | Mostly product and cart pages |
| Margin gap | Store orders and costs | Large, on many orders |
None of these proves the popup is a waste. Together they tell you whether it is worth testing, and how much is at stake if you do.
The test that answers it
Hold the popup back from a random share of new visitors, keep everything else the same, and compare the two groups on revenue and margin per visitor over a window long enough for the welcome emails to do their work. That comparison is a holdout test, and it is the only design here with a group that never saw the popup, a true control group.
- Randomize by visitor, if your tool allows it. If it does not, alternate weeks with and without the popup and run several cycles, so one good week cannot decide it.
- Judge on margin per visitor across all orders in the window, not on signup rate.
- Fix the traffic split by hand. Some tools shift traffic toward the leading variation while a test runs. Klaviyo's sign-up form tests do this unless you set the weights manually (Klaviyo Help Center). A split that moves in the middle of a test is much harder to read.
- If you cannot hide the popup, test the offer instead: the same popup with the discount against a newsletter-only version. It answers a narrower question, what the discount adds on top of the signup, but it answers it cleanly.
- Run it longer than your welcome flow, so buyers who come back on the last email are counted in both groups.
How long is long enough depends on your own weekly orders; how long an incrementality test should run walks through the sizing. The same logic settles gift offers too, as in free gift vs discount: what a holdout would show.
Who is counting
The popup tool reports signups, and the email tool reports the revenue of the people who signed up. Both numbers rise when you make the popup more aggressive, including when the extra signups are buyers who were already at checkout. Neither report has a control group, so neither can show what the popup adds. That is not a flaw in the tools. It is what they were built to count, and it is why Klaviyo will claim your Black Friday too.
Where a read fits
A popup test is a store-level question, and the holdout above answers it at no cost beyond the discount you already give. The channel-level version, whether email as a whole creates sales or collects them, is where a causal attribution tool like Causality Engine can help later: it reads one file, your GA4 Attribution paths export, and shows what each channel caused next to what last-click gave it. The read is EUR 99 once per upload, excluding VAT.
Related answers
- The Coupon and Affiliate Incrementality Audit: Stop Paying for Sales You Already Had
- How to Fix Klaviyo Over-Attributing Revenue on Shopify
- Ecommerce Discount Strategy: Do Your Promotions Actually Make Money?
- Klaviyo Will Claim Your Black Friday. How Much to Believe
- Free Gift vs Discount: What a Holdout Would Show
Frequently asked questions
How do I know if my welcome discount is incremental?
Hold the popup back from a random share of new visitors and compare revenue and margin per visitor between the two groups over a few weeks. Signup rate and the email tool's attributed revenue cannot answer it, because neither has a group that never saw the popup.Why does my welcome flow show so much revenue?
Email tools attribute an order to a message when it falls inside an attribution window after an open or a click. A buyer who opens the welcome email to copy the code on the way to checkout counts, even if they would have bought without it.What should I measure instead of signup rate?
Margin per visitor across everyone in the test, the share of first orders that use the code, and the time from signup to first order. Together they show whether the popup creates orders or discounts orders that were already coming.Can I test the discount without removing the popup?
Yes. Run the same popup with and without the discount, a newsletter-only version against the coded one, with the traffic split fixed by hand. It answers a narrower question: what the discount adds on top of the signup.
Go deeper: Causal attribution, explained.
Sixty-second versions of these ideas: Causality Engine on YouTube Shorts.
Keep reading
Terms in this article
- Attribution WindowAttribution Window is the defined period after a user interacts with a marketing touchpoint, during which a conversion can be credited to that ad. It sets the timeframe for assigning conversion credit.
- Black FridayBlack Friday is the day after Thanksgiving in the United States. It marks the start of the Christmas shopping season and is a major sales event for retailers.
- Causal AttributionCausal Attribution uses causal inference to determine which marketing touchpoints genuinely cause conversions, not just correlate with them.
- Control GroupControl Group is a segment of an audience intentionally not exposed to a marketing campaign, used to measure the campaign's true causal impact.
- Holdout TestA holdout test is an experiment where a portion of the audience does not see a campaign. This measures the campaign's true incremental impact.
- IncrementalityIncrementality measures the true causal impact of a marketing campaign. It quantifies the additional conversions or revenue directly from that activity.
- Product PagesProduct Pages are individual pages on an e-commerce site that describe a specific product. They provide detailed information to shoppers.
- Welcome EmailA Welcome Email is an automated email sent to new subscribers or customers. It introduces them to a brand and sets expectations for future communications.