How to run an email holdout test, step by step

Usually in nine steps inside Klaviyo. Pick one busy flow, split it at random right under the trigger, and leave one path empty. Label both paths with a profile property, let the test run past your slow buyers, then count who bought in each group. The gap is what the emails caused.

You need a Klaviyo flow with steady traffic, your store's orders arriving in Klaviyo as Placed Order events, and rights to edit flows. Building the test takes an hour. Waiting for it takes weeks, which is the part people skip.

Step by step

1. Pick one flow with steady traffic. Abandoned checkout, browse abandonment and post-purchase flows start on a trigger. Each new entrant can be sorted at random. Skip a flow you plan to rewrite this month. Path: Klaviyo > Flows tab > open the flow.
2. Add a split right under the trigger. Choose Random sample as its condition. Klaviyo's guide puts your percentage on the first path and suggests treating it as the control. Make that path the slice you hold out, such as one in ten. Path: Klaviyo > Flows > your flow > add a split > Random sample.
3. Leave the control path empty. Put the flow's existing emails, unchanged, on the second path. People on the first path get nothing from this flow, and everything else they receive stays as it was. Path: Klaviyo > Flows > your flow > the split's second path.
4. Label both paths. Drag a Profile property update action onto each path, above any email. Create a new property, email_test, set to holdout on the empty path and emailed on the other. Path: Klaviyo > Flows > your flow > Profile property update > + Add step > Create new property.
5. Tag the email links. Turn on UTM tracking for every email in the flow. GA4 can then show its own Email figure next to your test. GA4 files a session under Email only when its source or medium matches email. Path: Klaviyo > Flows > your flow > select a message > Settings > Enable UTM Tracking.
6. Write the decision rule, then wait. Note the start date, the end date and the gross profit the flow must add to stay. Then leave the flow alone until each group has a fair number of buyers. Path: Klaviyo > Flows > your flow > Show Analytics icon, to watch both paths fill.
7. Count the buyers in each group. For each label, build one segment with Properties about someone. Then build a second that adds What someone has done (or not done): Placed order, at least once, after your start date. Buyers divided by group size is that group's buying rate. Path: Klaviyo > Audience > Lists & segments > Create New > Create segment.
8. Work out the gap and price it. Subtract the holdout's buying rate from the emailed group's, then multiply by the emailed group's size. That is how many buyers the flow added; set it next to the conversions Klaviyo attributes to the flow for the same dates. Path: Klaviyo > Flows > your flow > Show Analytics icon.
9. Hold out campaigns too, if you qualify. Klaviyo's global holdout groups need at least 400,000 profiles, and only Owners, Admins and Managers can create one. Klaviyo recommends about 5% of marketable profiles, for 3 months. Path: Klaviyo > Analytics > Experiments > Global Experiments tab > Create test.

A worked example

Round, invented numbers, for illustration: an abandoned checkout flow over six weeks.

For illustration, the gap is 2.0 points, and 2.0% of the 9,000 emailed people is 180 added buyers. Say the flow report credits 540 conversions over the same dates. Klaviyo counts orders while the segments count people, so treat the comparison as rough: about one credited conversion in three was caused.

Now the money. In one store's Break-even sheet, a 40% margin gives a break-even ROAS of 2.5x, because 1 divided by 0.40 is 2.5. For illustration, at that margin and a €60 order, the 180 added buyers bring €10,800 of sales and €4,320 of gross profit. If the flow costs less than €4,320 to run over those six weeks, it pays its way. For illustration, the 540 credited conversions at €60 would suggest €32,400 of sales, a bar almost any flow clears.

Then the timing. In one store's Journeys sheet, journeys with 4 to 9 touches took 16.9 days to buy. If your slow buyers look like that, keep counting for about two and a half weeks after the last person enters. Stop sooner and the late buyers in both groups go missing.

Last, the noise. For illustration, 60 buyers in the holdout is not many. If the gap were 0.5 points instead of 2.0, a handful of buyers could flip it. You would run longer before cutting anything.

What to check when the report looks wrong

• Klaviyo credits far more than the gap. Expected. Its default windows count an order up to 5 days after an email open or click, whether or not the email changed anything.
• The holdout bought more than the emailed group. Check that the split sits directly under the trigger and that nobody edited the flow mid-test. If the result holds on a longer run, the flow may be costing sales, for example by teaching people to wait for a code.
• The control path stays empty. The split may be set to zero, which is how Klaviyo ends a test. Or a flow filter screens people out first.
• GA4 shows little or no Email revenue. Untagged clicks can fall into (direct) / (none). Accounts that saved Klaviyo UTM settings before 2 June 2025 may still send utm_medium=campaign or utm_medium=flow. GA4's Email rule ignores those unless the source says email.
• People switched groups. If someone enters the flow twice, the label shows the path they took last, because Klaviyo overwrites a property that already exists. If that worries you, test a flow people enter once, such as a welcome series.
• There is no Global Experiments tab. The account has fewer than 400,000 profiles, or your role is not Owner, Admin or Manager.
• The holdout still got order confirmations. That is by design. Transactional messages always reach a global holdout group, and a flow split only skips the flow it sits in.

What to do this week

1. Build the split today. In Klaviyo, open the Flows tab, pick your busiest flow, add a split under the trigger and choose Random sample. Pass: the first path holds your slice and contains no emails. Fail: emails sit on both paths, so move them before the flow goes live.
2. Save the four segments now. In Klaviyo, go to Audience, then Lists & segments, and create the label and buyer segments for both groups. Pass: both label segments start filling within a day of the flow going live. Fail: one stays empty, so the property name or value differs between the flow and the segment.
3. Put the decision rule in writing. Note the end date and the gross profit the flow must add to stay. Pass: the rule is dated and someone else has read it. Fail: you plan to decide by looking, which is how one lucky week becomes policy.

Check the homework. Your GA4 Attribution paths export already holds the evidence. Causality Engine reads that one file and shows what each channel caused next to what last-click gave it, in 1 to 2 minutes, for €99 once (excluding VAT), refundable within 30 days. Check the homework

Sources, 1 October 2026: How to A/B test flow branches (Klaviyo Help Center); How to add an update profile property action to a flow (Klaviyo Help Center); Getting started with segments (Klaviyo Help Center); Segmenting with dates reference (Klaviyo Help Center); How to add UTM tracking to campaigns and flows (Klaviyo Help Center); Understanding UTM tracking in Klaviyo (Klaviyo Help Center); Getting started with global holdout groups (Klaviyo Help Center); Understanding Klaviyo message attribution (Klaviyo Help Center); Default channel group (Google Analytics Help); Understand (direct) / (none) traffic (Google Analytics Help)

Related answers

• All Incrementality questions
• How do I test if email is incremental?
• Incrementality testing
• Holdout test, in the glossary