Causal Inference vs. LLM Attribution: What Each Can Identify
A comparison of two methods, not a score. What each can identify, what dated benchmarks measured for each, and four pass-or-fail checks you can run on your own export.
Three platforms, three answers, one budget to defend. Your GA4 export already holds the answer, and the first finding is free and never leaves your browser. Get your first finding free
By Joris van Huët, Founder & CEOUpdated 3 min read
They answer different questions, so a fair head-to-head needs a known right answer, and an experiment is how you get one. The published results are task-specific: in 663 Facebook experiments, two observational methods did not reliably recover the experimental effect, and in 2023 papers LLMs did well on one causal text task and close to random on another. None of them scores "LLM attribution" on marketing data.
What does each method need, and what can it identify?
| Causal inference | LLM asked to attribute | |
|---|---|---|
| Needs | A comparison: a holdout or geo test, or assumptions you can state | Whatever you paste, upload or connect, and a prompt |
| Can identify | What a change in spend did to revenue, in the regions and weeks tested, if the comparison is fair | Counts, summaries and explanations of the data it was given, plus the query behind them if you ask |
| Limit | Anything you didn't test, and anything the assumptions get wrong | Cause: data with no comparison group can't show it, whoever reads the data |
| Check | Did the two groups move together before the test? Repeat it once | Re-run it, run its query yourself, reconcile totals to Shopify |
What do the published results show?
Gordon, Moakler and Zettelmeyer (arXiv preprint, 2022) tested double/debiased machine learning and stratified propensity score matching against 663 Facebook experiments. Lewis and Rao (Quarterly Journal of Economics, 2015, peer-reviewed) add the other side: across 25 field experiments the median confidence interval on return on investment was "over 100 percentage points wide", so even an experiment can be too small to be informative.
Jin and colleagues (ICLR 2024, peer-reviewed) built the Corr2Cause benchmark, evaluated seventeen LLMs on deriving causal relations from correlational statements, and found performance "almost close to random". Both are text tasks on 2023 models. Models have changed since, so read them as a reason to test, not as a score.
OpenAI's API documentation says Chat Completions "are non-deterministic by default". No public measurement is cited here of how much an LLM's channel ranking varies between runs on attribution data, which is why the first check below exists.
How do you check each one yourself?
Four checks, in this order:
- Re-run it. Ask the same question in three fresh chats, or repeat the holdout once. Pass: the ranking or the lift comes out the same. Fail: it moves, so treat it as a draft.
- Ask for the comparison. What is the control group, or which assumption stands in for one? Pass: a named group or assumption. Fail: confident text with neither.
- Reproduce it. Get the query or the formula, run it on your own export and compare. Pass: same numbers. Fail: any difference you can't trace.
- Reconcile it. Orders and revenue should match Shopify's for the same dates and time zone. Pass: they match. Fail: a gap you can't name.
Rule of thumb: let an LLM draft and explain, and let a comparison decide.
Sources, 30 September 2026: Close Enough? (arXiv, 2022); The Unfavorable Economics of Measuring the Returns to Advertising (Quarterly Journal of Economics, 2015); Causal Reasoning and Large Language Models (arXiv, 2023, revised 2024); Can Large Language Models Infer Causation from Correlation? (ICLR 2024); How is ChatGPT's behavior changing over time? (arXiv, 2023); Advanced usage (OpenAI API documentation).
Related answers
Frequently asked questions
Is causal inference more accurate than LLM attribution?
They answer different questions. A holdout or geo test estimates what spend caused in the regions and weeks you tested; an LLM drafts analysis from the data you give it. To compare them fairly, score both against the result of a holdout.Can I use an LLM to run a causal analysis?
It can help set one up: Kıcıman and colleagues write that LLMs may be used by experts "to save effort in setting up a causal analysis". The estimate itself should come from your data through a method you can re-run, such as a holdout.How do I check an LLM's attribution answer?
Ask the same question in three fresh chats, ask for the query, run it on your export and reconcile the totals to Shopify. If the channel ranking changes between runs, treat the answer as a draft.
Go deeper: Causal attribution, explained.
Sixty-second versions of these ideas: Causality Engine on YouTube Shorts.
Have an audience of ecommerce operators? Earn 20% recurring commission by recommending the €99 read.
Keep reading
Terms in this article
- Attribution PlatformAttribution Platform is a software tool that connects marketing activities to customer actions. It tracks touchpoints across channels to measure campaign impact.
- Causal InferenceCausal inference is the set of methods for estimating how much an outcome changes because of a cause, rather than alongside it. It works by comparing what happened with an estimate of what would have happened without the cause.
- Confidence IntervalConfidence Interval is a statistical range of values that likely contains the true value of a metric. In marketing analytics, it quantifies uncertainty around estimates, indicating the precision of an outcome or causal effect.
- CounterfactualA counterfactual is what would have happened under conditions that did not occur, such as a customer's outcome had they not received the campaign they did receive. It is never observed, so every effect estimate has to stand in for it with a control group, comparable units or a forecast.
- Incrementality TestingIncrementality Testing measures the additional impact of a marketing campaign. It compares exposed and control groups to determine causal effect.
- Machine LearningMachine Learning involves computer algorithms that improve automatically through experience and data. It applies to tasks like customer segmentation and churn prediction.
- Propensity ScoreA propensity score is the probability a unit receives a specific treatment given observed characteristics. It reduces selection bias in observational studies, enabling causal inference when randomized experiments are not possible.
- Propensity Score MatchingPropensity Score Matching is a statistical method that estimates the causal effect of a treatment from observational data. It matches individuals with similar likelihoods of receiving treatment to isolate its impact.