Can LLMs Tell Correlation From Causation in Attribution?
Not from your data alone. One published test found seventeen LLMs near random at inferring causation from correlation statements, and observational methods often failed to match ad experiments. A holdout test is how you check.
Three platforms, three answers, one budget to defend. Your GA4 export already holds the answer, and the first finding is free and never leaves your browser. Get your first finding free
By Joris van Huët, Founder & CEOUpdated 4 min read
Not from your data alone, and the published tests show why. On Corr2Cause (ICLR 2024), seventeen LLMs performed "almost close to random" at inferring causation from correlational statements. In 15 Facebook advertising experiments (Gordon et al., Marketing Science, 2019), observational methods often failed to reproduce the randomized result.
What did the two LLM tests measure?
Corr2Cause (Jin et al., ICLR 2024, peer-reviewed) gives a model a set of correlational statements and asks it to determine "the causal relationship between the variables". The authors built more than 200K samples and evaluated seventeen LLMs. They report that the models "achieve almost close to random performance", and that fine-tuned models "fail to generalize" when variable names and wording change.
Kiciman and colleagues (TMLR, peer-reviewed) asked a different question: can a model use what it has read to answer causal questions about named variables? The authors say this works "by operating on the text metadata", that "LLMs exhibit unpredictable failure modes", and that "LLMs ignore the actual data". Both tested the models of their time, not today's.
What does field evidence say about observational data?
Gordon and colleagues (Marketing Science, 2019, peer-reviewed; two of the four authors were at Facebook) used 15 U.S. advertising experiments at Facebook, with 500 million user-experiment observations and 1.6 billion ad impressions, to compare randomized results with those of multiple observational models. "The observational methods often fail to produce the same effects as the randomized experiments, even after conditioning on extensive demographic and behavioral variables."
A randomized design supplies that variation; a table of past exposure does not. An LLM with a code tool can run whatever estimator you name, but the variation has to come from a design or from assumptions you state and check. No source cited here tested an LLM on estimating ad effects from a store's own data.
How do you check a causal claim against your own data?
Turn the claim into a test in four steps:
- Write it as a counterfactual. "Without this spend, these weeks' orders would have been lower by about X." If you cannot say what the comparison is, the claim is only a correlation.
- Check whether spend follows demand. Line spend up against your own calendar of launches, sales and holidays. For illustration: if you set each week's spend at a tenth of the revenue you expect, spend of 10, 10, 10, 16, 18 and 20 tracks revenue of 100, 100, 100, 160, 180 and 200 perfectly (correlation 1.0), even if the ads added nothing.
- Run a small holdout. Pick comparable regions, change spend in some of them for a period you fix in advance, and keep it unchanged in the rest. Google researchers described this design in 2011: regions "randomly assigned to a control or treatment condition". Export orders by region from Shopify for the same weeks.
- Compare the change, not the level. Measure how orders moved in the regions where spend changed, relative to the regions where it stayed the same, over the same weeks. Then compare that gap with how much those regions differed week to week before the change.
Pass: the gap is larger than the usual week-to-week differences, so the change in spend moved orders by more than noise. Fail: the gap sits inside that range. The test cannot tell you either way, and a longer test or a bigger change is the next step. Fix the length and the threshold before you look.
Where does a causal read fit?
Sources, 30 September 2026: Can Large Language Models Infer Causation from Correlation? (Jin et al., ICLR 2024); Causal Reasoning and Large Language Models: Opening a New Frontier for Causality (Kiciman et al., TMLR; arXiv, 2023); A Comparison of Approaches to Advertising Measurement: Evidence from Big Field Experiments at Facebook (Gordon et al., Marketing Science, 2019); Measuring Ad Effectiveness Using Geo Experiments (Vaver and Koehler, Google, 2011).
Related answers
Frequently asked questions
Can an LLM do causal inference?
It depends on the task. On Corr2Cause, seventeen LLMs performed close to random at inferring causation from correlation statements, while GPT-3.5 and GPT-4 scored well on causal questions answered from variable names. Neither test estimated an effect from a store's own data, which needs a design or stated assumptions.What is the difference between correlation and causation in marketing?
Correlation means two numbers move together, such as spend and orders. Causation means orders would have been different without the spend. A holdout or geo test is how you separate the two.How do I know if my ad spend caused the sales?
Compare orders where spend changed with orders where it did not, over the same weeks. A holdout region is the simplest design. A dashboard or an LLM summary cannot settle it alone, because the orders you would have had without the spend are not in your data.
Go deeper: Incrementality testing, explained.
Sixty-second versions of these ideas: Causality Engine on YouTube Shorts.
Have an audience of ecommerce operators? Earn 20% recurring commission by recommending the €99 read.
Keep reading
Terms in this article
- Ad ImpressionAd Impression is a single instance of an advertisement displaying on a webpage. Impressions are a key input for models measuring the causal impact of ad exposure on user behavior.
- AttributionAttribution identifies user actions that contribute to a desired outcome and assigns value to each. It reveals which marketing touchpoints drive conversions.
- Causal AttributionCausal Attribution uses causal inference to determine which marketing touchpoints genuinely cause conversions, not just correlate with them.
- ConfoundingConfounding is a distortion of the estimated treatment effect when a third variable, a confounder, associates with both the treatment and the outcome. Causal inference methods control for confounding to isolate the true treatment effect.
- Confounding VariableConfounding Variable is an unmeasured factor that influences both the marketing input and the desired outcome, distorting the true impact of a campaign.
- CounterfactualA counterfactual is what would have happened under conditions that did not occur, such as a customer's outcome had they not received the campaign they did receive. It is never observed, so every effect estimate has to stand in for it with a control group, comparable units or a forecast.
- IncrementalityIncrementality is the part of an outcome, such as orders or revenue, that a marketing activity caused: the extra results compared with what would have happened without it. It is a causal effect, not the number of conversions credited to a channel.
- Incrementality TestingIncrementality Testing measures the additional impact of a marketing campaign. It compares exposed and control groups to determine causal effect.