Vary one element, learn something reusable
Creative tests that change several things at once produce a winner and no transferable knowledge. Isolating one element is what makes the result apply to the next thing you make.
Three platforms, three answers, one budget to defend. Your GA4 export already holds the answer, and the first finding is free and never leaves your browser. Get your first finding free
By Joris van Huët, Founder & CEOUpdated 2 min read
A test that changes five things tells you which of two images won, and nothing you can apply to the sixth image. Isolation is what makes a result reusable.
Winner versus knowledge
Two creatives, entirely different, one wins. You now know which to run. You know nothing about why, so the next pair is another coin flip.
Two creatives differing only in headline, one wins. You now know something about headlines that applies to everything you make next.
The second is worth more even when the first produces a bigger immediate lift.
What counts as one element
| One element | Not one element |
|---|---|
| Headline text, same layout and image | A different concept entirely |
| Image background, same subject and copy | New image and new copy |
| Price point, same page | Price plus a new badge |
| First line of the description | Rewritten description |
The right-hand column is not wrong to run. It is just a choice of a winner rather than a lesson, and it should be labelled as such so nobody later cites it as evidence about headlines.
The exception worth knowing
When you have no idea what works at all, a wide test that varies everything is a reasonable first move. It finds the rough region quickly.
Then narrow. The mistake is staying in wide-test mode forever and wondering why the learnings never compound.
Sample size before you start
Decide how many conversions constitute a readable result before running, and stop at that point rather than when a difference appears. Watching a test until it looks significant is how noise gets promoted to a finding.
The read
A causal read on a Google Analytics export shows what each channel caused next to what last-click gave it, with a data-health score rather than a confidence interval, for 99 euro once, refundable within 30 days, no questions asked. Where the design cannot support a causal claim it says so instead of producing a number.
Related answers
Frequently asked questions
Why test one element at a time?
A test of two entirely different creatives tells you which won and nothing about why, so the next pair is another coin flip. A test that isolates the headline teaches you something that applies to everything you make next.Is a wide test ever right?
Yes, when you have no idea what works at all. A wide test finds the rough region quickly. The mistake is staying in wide-test mode permanently and wondering why nothing compounds.When should I stop a creative test?
At the conversion count you decided on before starting. Watching until a difference appears is how noise gets promoted to a finding.
Go deeper: Causal attribution, explained.
Sixty-second versions of these ideas: Causality Engine on YouTube Shorts.
Have an audience of ecommerce operators? Earn 20% recurring commission by recommending the €99 read.
Keep reading
Terms in this article
- AnalyticsAnalytics is the systematic computational analysis of data. It reveals customer behavior and measures campaign performance.
- AttributionAttribution identifies user actions that contribute to a desired outcome and assigns value to each. It reveals which marketing touchpoints drive conversions.
- Confidence IntervalConfidence Interval is a statistical range of values that likely contains the true value of a metric. In marketing analytics, it quantifies uncertainty around estimates, indicating the precision of an outcome or causal effect.
- ConversionConversion is a specific, desired action a user takes in response to a marketing message, such as a purchase or a sign-up.
- Google AnalyticsGoogle Analytics is a web analytics service that tracks and reports website traffic.
- HeadlinesHeadlines are the titles of articles, advertisements, or webpages. They capture attention and encourage further engagement.
- IncrementalityIncrementality is the part of an outcome, such as orders or revenue, that a marketing activity caused: the extra results compared with what would have happened without it. It is a causal effect, not the number of conversions credited to a channel.
- Product PageProduct Page is a webpage dedicated to a single product. It includes images, descriptions, pricing, and purchase options.