Incrementality Testing
Incrementality Testing measures the additional impact of a marketing campaign. It compares exposed and control groups to determine causal effect.
Three platforms, three answers, one budget to defend. Your GA4 export already holds the answer, and the first finding is free and never leaves your browser. Get your first finding free
By Joris van Huët, Founder & CEOUpdated 4 min read
Run the numbers for your store: the free holdout test planner.
What is Incrementality Testing?
Incrementality Testing is a rigorous analytical method used to determine the true causal impact of marketing campaigns by isolating the additional value generated beyond what would have happened organically. Unlike traditional attribution models that allocate credit based on predefined rules or multi-touch heuristics, incrementality testing employs controlled experiments, such as randomized control trials (RCTs), to compare outcomes between groups exposed to the marketing intervention and control groups not exposed. This approach helps ecommerce brands precisely identify whether specific campaigns contribute to incremental sales or customer acquisitions, rather than merely measuring correlation or engagement metrics.
Historically, incrementality testing emerged from the need to overcome attribution challenges created by complex customer journeys and cross-channel marketing effects. In ecommerce, where consumers often interact with multiple touchpoints like paid social ads, email campaigns, and influencer endorsements, it can be difficult to decipher the true effectiveness of each channel.
Technically, incrementality tests involve splitting the target audience into two statistically similar groups: a test group that receives the marketing treatment and a control group that does not. By measuring key performance indicators (KPIs) such as revenue, conversion rate, or customer lifetime value across these groups, marketers can quantify the lift attributable solely to the campaign. This method ensures investment decisions are data-driven and grounded in causality rather than assumptions or last-click attribution biases.
Why Incrementality Testing matters for ecommerce
For ecommerce marketers, incrementality testing is important because it directly informs budget allocation and strategy improvement by revealing which campaigns genuinely drive incremental revenue and customer growth. Incrementality testing also improves ROI measurement accuracy. By quantifying the net lift from marketing efforts, brands can confidently scale high-performing campaigns that generate positive returns and cut back on those with little or no incremental impact. Beyond financial metrics, incrementality testing provides a competitive advantage by enabling data-driven growth. Brands that adopt incrementality frameworks can continuously experiment, learn from controlled insights, and refine messaging or targeting with confidence. This scientific approach to marketing attribution reduces guesswork, enhances transparency with stakeholders, and accelerates sustainable growth in the evolving ecommerce ecosystem.
How to use Incrementality Testing
- Define Clear Objectives: Start by selecting specific KPIs to measure, such as incremental sales, new customers acquired, or average order value. For example, a Shopify store can focus on monthly revenue lift from a new Instagram ad campaign.
- Segment Your Audience: Use customer data to create statistically comparable test and control groups. This can be done via geo-splits, holdout audiences, or randomized user IDs. Ensure groups are balanced to minimize bias.
- Deploy Campaigns Selectively: Run your marketing campaign only on the test group while withholding it from the control group during the test period.
- Collect and Analyze Data: Track conversions and revenue from both groups.
- Interpret Results: Calculate the incremental lift by comparing KPIs between groups. Assess statistical significance to confirm findings.
- Act on Insights: Scale campaigns with proven incrementality and pause or improve those without. Continuously run incremental tests to refine marketing mix.
Formula
Incrementality (%) = ((Conversion Rate in Test Group - Conversion Rate in Control Group) / Conversion Rate in Control Group) × 100
Frequently asked questions
What is the difference between incrementality testing and traditional attribution?
Incrementality testing measures the true causal impact of a marketing campaign by comparing exposed and control groups, isolating the additional value generated. Traditional attribution models assign credit based on touchpoints but can’t distinguish between correlation and causation, often leading to overestimated campaign performance.How can e-commerce brands implement incrementality testing without disrupting customers?
Brands can use holdout groups or geo-based splits to exclude a small, representative segment from campaigns, minimizing disruption.What types of campaigns benefit most from incrementality testing?
Campaigns with complex customer journeys, such as multi-channel paid ads, influencer partnerships, and retargeting efforts, benefit significantly. Incrementality testing clarifies which efforts truly drive new conversions versus those merely capturing existing demand.Can incrementality testing help optimize ad spend across platforms?
Yes, by quantifying the incremental impact of each channel, marketers can allocate budgets more effectively, scaling high-performing campaigns and reducing spend on those with low or no incremental lift, thereby improving overall ROI.