Can an LLM Tell If an Attribution Result Is Significant?
Only if you hand it two groups to compare, and you should still recompute. A published benchmark found LLMs primarily err on choosing a valid test; four counts and one formula let you check the verdict in a spreadsheet.
Three platforms, three answers, one budget to defend. Your GA4 export already holds the answer, and the first finding is free and never leaves your browser. Get your first finding free
By Joris van Huët, Founder & CEOUpdated 5 min read
Only if it is given two groups to compare, and even then the verdict is narrower than it sounds. Across twenty-five large field experiments (Lewis and Rao, Quarterly Journal of Economics, 2015), the median confidence interval on return on investment was over 100 percentage points wide, so a test that is not significant may only have lacked the data. The test itself needs four counts and one formula, so you can rerun any verdict in a spreadsheet.
What does a significant result tell you?
A significance test asks how unusual your data would be if there were no real difference, judged against a cut-off fixed in advance. The NIST/SEMATECH e-Handbook of Statistical Methods (official reference) builds it as a risk: "The test procedure is constructed so that the risk of rejecting the null hypothesis, when it is in fact true, is small."
A significant result is not the probability that the effect is real. Ioannidis (PLOS Medicine, 2005, a peer-reviewed essay with simulations) argues that the probability a research claim is true "may depend on study power and bias, the number of other studies on the same question, and, importantly, the ratio of true to no relationships among the relationships probed in each scientific field". A finding is less likely to be true, he writes, "where there is greater flexibility in designs, definitions, outcomes, and analytical modes". Asking a model to "find what is significant" hands it that flexibility.
How noisy is ad measurement in practice?
Lewis and Rao (peer-reviewed) analysed twenty-five large field experiments with major U.S. retailers and brokerages, most reaching millions of customers. They report that the median confidence interval on return on investment is over 100 percentage points wide, that "a coefficient of variation of 10 is common" in individual-level sales, and that informative experiments "can easily require more than 10 million person-weeks". A test on a smaller audience has less data, so "not significant" tells you even less.
Can an LLM run the test for you?
StatQA (Zhu et al., NeurIPS 2024, peer-reviewed) has 11,623 examples built to test the applicability of statistical methods, particularly hypothesis tests. The authors report that "LLMs primarily make applicability errors": choosing a method whose assumptions do not fit the data type, distribution or sample size. They note that computation "can be easily aided by external tools". That is a 2024 snapshot of the models tested, and no source cited here tested an LLM on attribution significance.
How do you run the test yourself?
You need a holdout or A/B split, with visitors and orders counted in each group:
- Fix the cut-off and the test length before you look. Write both down.
- Compute each group's rate and the pooled rate. Orders divided by visitors for each group, and total orders divided by total visitors for the pool.
- Compute z with the large-sample test for two proportions in the NIST Handbook.
- Turn z into a two-sided p-value with the formula on the last line below, which works in Google Sheets.
- Compare p with your cut-off.
rate_A = orders_A / visitors_A
rate_B = orders_B / visitors_B
pooled = (orders_A + orders_B) / (visitors_A + visitors_B)
z = (rate_A - rate_B) / SQRT(pooled * (1 - pooled) * (1 / visitors_A + 1 / visitors_B))
p = 2 * (1 - NORMSDIST(ABS(z), TRUE))
Pass: p is below the cut-off you set, so a gap this large would be unusual if the groups were really the same. Then look at its size and its cost, because significance says nothing about how big the effect is. Fail: p is above the cut-off. The data are compatible with no difference, which is not the same as no difference.
For illustration: with 2,000 visitors per group, 100 control orders and 120 test orders give rates of 5.0% and 6.0%, a pooled rate of 220 / 4,000 = 5.5%, a standard error of sqrt(0.055 x 0.945 x 2 / 2,000) = 0.0072 and z = 0.01 / 0.0072 = 1.39, a two-sided p-value of about 0.17. For illustration: the same 5.0% and 6.0% with 10,000 visitors per group (500 and 600 orders) give a standard error of 0.0032, z = 0.01 / 0.0032 = 3.1 and a p-value of about 0.002. Same lift, different verdict: the amount of data changed, not the effect.
Then give the model the same four counts, ask for z and the p-value, and compare with yours. If they differ, find which step differs before trusting either. An LLM is good at writing the spreadsheet formula and explaining the NIST page in plain words.
Sources, 30 September 2026: The Unfavorable Economics of Measuring the Returns to Advertising (Lewis and Rao, Quarterly Journal of Economics, 2015); Are Large Language Models Good Statisticians? (Zhu et al., NeurIPS 2024; full text); Why Most Published Research Findings Are False (Ioannidis, PLOS Medicine, 2005); NIST/SEMATECH e-Handbook, 7.3.3 two proportions and 7.1.3 statistical tests (NIST); NORMSDIST (Google Docs Editors Help).
Related answers
Frequently asked questions
What does statistically significant mean in marketing?
It means the gap between two groups would be unusual if chance alone were at work, judged against a cut-off you set beforehand. It is not the probability that the channel works, and it says nothing about how big the effect is.Can an LLM calculate statistical significance for my A/B test?
It can apply the formula, and with a code tool it can run it. A peer-reviewed 2024 benchmark found the harder part is choosing a test whose assumptions fit the data. Give it your four counts and compare its z and p-value with a spreadsheet.My test is not significant. Does that mean the channel does not work?
No. A large p-value means the data are compatible with no difference, and NIST notes that a non-result may indicate you do not yet have enough data. Check how many visitors each group had before you cut spend.
Go deeper: Incrementality testing, explained.
Sixty-second versions of these ideas: Causality Engine on YouTube Shorts.
Have an audience of ecommerce operators? Earn 20% recurring commission by recommending the €99 read.
Keep reading
Terms in this article
- AttributionAttribution identifies user actions that contribute to a desired outcome and assigns value to each. It reveals which marketing touchpoints drive conversions.
- CausationCausation is the relationship where a change in one variable directly causes a change in another.
- Confidence IntervalConfidence Interval is a statistical range of values that likely contains the true value of a metric. In marketing analytics, it quantifies uncertainty around estimates, indicating the precision of an outcome or causal effect.
- CorrelationCorrelation is a statistical measure showing a relationship between variables; it does not imply causation.
- ExperimentsExperiments are scientific procedures that test hypotheses or demonstrate facts. In marketing, experiments like A/B tests determine the causal effect of campaign changes, enabling data-driven decisions.
- IncrementalityIncrementality is the part of an outcome, such as orders or revenue, that a marketing activity caused: the extra results compared with what would have happened without it. It is a causal effect, not the number of conversions credited to a channel.
- Incrementality TestingIncrementality Testing measures the additional impact of a marketing campaign. It compares exposed and control groups to determine causal effect.
- Statistical SignificanceStatistical Significance measures the probability that observed results are not due to random chance. It confirms the reliability of test outcomes.