Attribution Requires Causal Models, Not Language Models
Orders alone can't show what a channel caused. What the published evidence says about observational methods, where an LLM helps, and a region holdout you can run from a Shopify export.
By Joris van Huët, Founder & CEOUpdated 4 min read
Attribution asks what would have happened without the spend, and orders alone can't answer that, whichever tool reads them. In 15 Facebook ad experiments, observational methods often failed to reproduce the randomized results (Marketing Science, 2019), and Shopify says its own "Any click" model "allocates more credit than orders you've received". Credit is a rule for splitting sales. Cause needs a holdout or assumptions you can state.
What does the published evidence say about reading cause from observed data?
Gordon, Zettelmeyer, Bhargava and Chapsky (Marketing Science, 2019, peer-reviewed) compared 15 US advertising experiments at Facebook, with 500 million user-experiment observations and 1.6 billion ad impressions, against multiple observational models. Their result: "The observational methods often fail to produce the same effects as the randomized experiments, even after conditioning on extensive demographic and behavioral variables."
Gordon, Moakler and Zettelmeyer (arXiv preprint, 2022) repeated the exercise on 663 Facebook experiments with over 5,000 user-level features.
Both studies use Facebook's ads and data. Neither tests an LLM, and no public measurement is cited here of how an LLM's attribution answer compares with an experiment.
Where does the missing comparison come from?
From a design. Google's help page says its data-driven attribution models "compare the key event probability of users who were exposed to the ad, to the key event probability of similar users in a holdback group", and it defines attribution as "the act of assigning credit". Change the rule and the credit moves: GA4's models exclude direct visits from credit unless the whole path is direct, while Shopify offers a last-click model with direct visits and another without them. A tool that can't say where its comparison comes from is reporting credit.
What can a language model add to a causal analysis?
A drafting aid, on the evidence. Use an LLM to list what else moved in your test weeks, such as a promotion, a price change or a stock-out, then check each item against your calendar.
How do you run a region holdout from your Shopify export?
Five steps, using the Created at, Total and Shipping Province Name columns of an orders export:
- Split the regions. Make two sets that sold similar amounts in the same recent weeks. Pass: their weekly revenue moves together. Fail: it drifts apart, so pick other regions.
- Change one set only. Switch the channel off, or cut it a little, in one set. Leave prices, promotions and other channels alone, and write the start date down.
- Give it enough days. Lewis and Rao (Quarterly Journal of Economics, 2015) ran 25 field experiments with US retailers and brokerages and report that the median confidence interval on return on investment was "over 100 percentage points wide". The holdout test planner counts the days your order volume needs.
- Compare the change in revenue between the two sets. Pass: the set with less spend fell behind by more than the weekly gaps you saw before the change. Fail: no gap beyond that noise, which means either the channel wasn't causing those sales or the test was too small to show it.
- Repeat it once before you move a large budget.
Where does a causal read fit?
Sources, 30 September 2026: A Comparison of Approaches to Advertising Measurement (Marketing Science, 2019); Close Enough? (arXiv, 2022); The Unfavorable Economics of Measuring the Returns to Advertising (Quarterly Journal of Economics, 2015); Get started with attribution (Google Analytics Help); Marketing reports (Shopify Help Center); Exporting orders (Shopify Help Center); Causal Reasoning and Large Language Models (arXiv, 2023).
Related answers
Frequently asked questions
Can an LLM do causal attribution?
Not from your orders alone. An LLM can draft the test plan and list what else moved, but the effect has to come from a comparison in your data, such as a holdout. Kıcıman and colleagues write that LLMs "ignore the actual data" when they build causal arguments from text.What's the difference between attribution and causation?
Attribution assigns credit to touchpoints by a rule: Google's help defines it as "the act of assigning credit for important user actions". Causation asks what would have happened without the spend, which needs a comparison such as a holdout or a geo test.How do I know if an attribution tool is causal?
Ask where its comparison comes from: a holdout, a geo test, or assumptions it states. If it names none of these, it is reporting credit. Even Google's data-driven model describes itself as using a holdback group.
Go deeper: Incrementality testing, explained.
Sixty-second versions of these ideas: Causality Engine on YouTube Shorts.
Have an audience of ecommerce operators? Earn 20% recurring commission by recommending the €99 read.
Keep reading
Terms in this article
- Attribution ModelAn Attribution Model defines how credit for conversions is assigned to marketing touchpoints. It dictates how marketing channels receive credit for sales.
- Causal AnalysisCausal Analysis identifies true cause-and-effect relationships in data, moving beyond correlation to show how marketing actions directly impact outcomes.
- Causal AttributionCausal Attribution uses causal inference to determine which marketing touchpoints genuinely cause conversions, not just correlate with them.
- Causal InferenceCausal inference is the set of methods for estimating how much an outcome changes because of a cause, rather than alongside it. It works by comparing what happened with an estimate of what would have happened without the cause.
- Confidence IntervalConfidence Interval is a statistical range of values that likely contains the true value of a metric. In marketing analytics, it quantifies uncertainty around estimates, indicating the precision of an outcome or causal effect.
- Google AnalyticsGoogle Analytics is a web analytics service that tracks and reports website traffic.
- IncrementalityIncrementality is the part of an outcome, such as orders or revenue, that a marketing activity caused: the extra results compared with what would have happened without it. It is a causal effect, not the number of conversions credited to a channel.
- Incrementality TestingIncrementality Testing measures the additional impact of a marketing campaign. It compares exposed and control groups to determine causal effect.