The Cost of Wrong Attribution: What Happens When You Trust an LLM with Budget Decisions
It costs whatever you moved on the strength of the number. The net on each extra euro is incremental ROAS times margin, minus one; here it is worked with your own numbers, with the field evidence on how far off the number can be.
Three platforms, three answers, one budget to defend. Your GA4 export already holds the answer, and the first finding is free and never leaves your browser. Get your first finding free
By Joris van Huët, Founder & CEOUpdated 4 min read
It costs whatever you moved on the strength of the number, and field experiments show that number can sit far from what the spend caused. In field experiments at eBay, brand keyword ads had no measurable short-term benefits. For illustration: at a 40% contribution margin before ad cost, break-even ROAS is 1 divided by 0.40, which is 2.5, so a channel that claims 4.0 while a holdout shows 2.0 loses EUR 0.20 on every EUR 1 you add to it, because 2.0 times 0.40 is 0.80.
What do field experiments say about the gap?
Three sets of field experiments, each published in a peer-reviewed journal, each with limits:
- eBay paid search. Blake, Nosko and Tadelis (Econometrica, 2015) report results from large-scale field experiments at eBay: "returns from paid search are a fraction of non‐experimental estimates", and brand keyword ads "have no measurable short‐term benefits". Scope: eBay's paid search, not your channels.
- Facebook. Gordon, Zettelmeyer, Bhargava and Chapsky (Marketing Science, 2019) used 15 US advertising experiments, with 500 million user-experiment observations and 1.6 billion ad impressions. Their finding: the observational methods "often fail to produce the same effects as the randomized experiments".
- Noise. Lewis and Rao (Quarterly Journal of Economics, 2015) report on 25 field experiments with major US retailers and brokerages, together $2.8 million of digital advertising spend: "The median confidence interval on return on investment is over 100 percentage points wide."
No public measurement was found of the average gap between platform-reported and incremental ROAS across online stores. Read the studies as evidence that the gap can be large, and as no evidence about the size of yours.
What does the gap cost?
Two sums, with your own numbers. The net on each extra euro is the incremental ROAS times your margin, minus one.
- Over-credited. For illustration: adding EUR 10,000 at a claimed 4.0 promises EUR 40,000 of revenue, while an incremental 2.0 delivers EUR 20,000. For illustration, at a 40% margin that is EUR 8,000 of contribution, so the EUR 10,000 you added loses EUR 2,000.
- Under-credited. For illustration: cutting EUR 10,000 from a channel with an incremental ROAS of 3.0 at a 40% margin saves EUR 10,000 and removes EUR 12,000 of contribution (3.0 × 0.40 × 10,000), a net loss of EUR 2,000.
Add spend where incremental ROAS times margin is above one, cut where it is below one, and hold where the test can't tell you. Experiments are noisy, so run both sums at each end of the test's range, not only at the point estimate.
How do you check the inputs with your own data?
- Margin and revenue on one basis. Use the same revenue definition for ROAS and margin. A Shopify orders export keeps Subtotal (before shipping and taxes) and Total in separate columns, and GA4's purchase revenue is net of refunds. The break-even ROAS calculator does the division.
- The claimed side. Add up what every platform claims for one closed week and compare it with total store revenue for the same week. For illustration: platforms claiming EUR 60,000 and EUR 50,000 against EUR 80,000 of store revenue are claiming EUR 30,000 more than the store took. If the claims add up to more than the store took, some orders are claimed twice or valued on a different basis.
- The incremental side. Switch the channel down or off in some regions for a set period and compare total revenue and orders there with the regions where it stayed on. Size the cut like the change you plan: switching a channel off measures its average return, not the return on the next euro. Pass: the regions were comparable and no promotion or stock-out hit one side only. Fail: they weren't, so the number isn't usable. The holdout test calculator shows how many days it must run.
An LLM can run these sums and draft the revenue query; check its arithmetic by hand once. What it can't change is the input: tracked conversions show what was credited, and the incremental ROAS has to come from a test, whoever does the arithmetic. No public measurement was found of how budgets set by an LLM perform against budgets set by an analyst.
Sources, 30 September 2026: Consumer Heterogeneity and Paid Search Effectiveness (Blake, Nosko and Tadelis, Econometrica, 2015); A Comparison of Approaches to Advertising Measurement (Kellogg School of Management, Marketing Science 2019); The Unfavorable Economics of Measuring the Returns to Advertising (Lewis and Rao, Quarterly Journal of Economics, 2015); Analytics dimensions and metrics (Google, 2026); Export orders (Shopify, 2026).
Frequently asked questions
How do I calculate the cost of wrong attribution?
Take the extra spend and the gap between the ROAS a platform claims and the incremental ROAS a holdout gives. The net on each extra euro is incremental ROAS times margin, minus one. Below zero, that spend loses money at the margin.What is break-even ROAS?
One divided by your contribution margin before ad cost, on the same revenue basis as the ROAS. For illustration: at a 40% margin it is 2.5.Can an LLM calculate ROAS for me?
It can do the arithmetic and draft the revenue query; check both against a Shopify export.
Go deeper: Incrementality testing, explained.
Sixty-second versions of these ideas: Causality Engine on YouTube Shorts.
Have an audience of ecommerce operators? Earn 20% recurring commission by recommending the €99 read.
Keep reading
Terms in this article
- Ad ImpressionAd Impression is a single instance of an advertisement displaying on a webpage. Impressions are a key input for models measuring the causal impact of ad exposure on user behavior.
- AttributionAttribution identifies user actions that contribute to a desired outcome and assigns value to each. It reveals which marketing touchpoints drive conversions.
- Confidence IntervalConfidence Interval is a statistical range of values that likely contains the true value of a metric. In marketing analytics, it quantifies uncertainty around estimates, indicating the precision of an outcome or causal effect.
- ConversionConversion is a specific, desired action a user takes in response to a marketing message, such as a purchase or a sign-up.
- ExperimentsExperiments are scientific procedures that test hypotheses or demonstrate facts. In marketing, experiments like A/B tests determine the causal effect of campaign changes, enabling data-driven decisions.
- Holdout TestA holdout test is an experiment that keeps a random part of an audience from seeing a campaign and compares its results with those of the part that could see it. The held-out part is the control group, so the gap in outcomes estimates what the campaign added.
- IncrementalityIncrementality is the part of an outcome, such as orders or revenue, that a marketing activity caused: the extra results compared with what would have happened without it. It is a causal effect, not the number of conversions credited to a channel.
- Incrementality TestingIncrementality Testing measures the additional impact of a marketing campaign. It compares exposed and control groups to determine causal effect.