Ten Questions to Ask Any Attribution Tool, LLM or Not
Ask which rule produced the number, then reconcile its totals to Shopify and GA4, rerun it, and ask what test would change its mind. Studies of ad experiments show last-click credit and observational methods often miss the experimental result.
Three platforms, three answers, one budget to defend. Your GA4 export already holds the answer, and the first finding is free and never leaves your browser. Get your first finding free
By Joris van Huët, Founder & CEOUpdated 4 min read
Start with the method: ask which rule produced the number. The gap between a credit rule and a causal result is measured, and it is large: across 2,226 Meta ad experiments, 7-day last-click attribution gave an out-of-sample R-squared of 0.19 against incremental conversions per dollar (Gordon, Moakler and Zettelmeyer, arXiv preprint, revised April 2026). The ten questions below check the method, the arithmetic and the causal claim, and each has a check you can run in an afternoon.
What do measurements say about attribution answers?
Two studies by the same authors test what advertisers' non-experimental data can recover. In the preprint above, last-click attribution is the baseline against which the authors' own method is compared. "Close Enough?" (arXiv, 2022; Marketing Science, 2023, peer-reviewed) analysed 663 large-scale experiments at Facebook. The authors conclude that, even with rich user-level data, they are unable to reliably estimate an ad campaign's causal effect. Neither study tests an LLM. Both test what observational data recovers, which is what a GA4 export contains.
Which ten questions should you ask any tool?
- Which rule or method produced this number? Check: it names one. GA4's Attribution reports offer three models, and all of them leave direct visits out of the credit unless the whole path is direct.
- Which dates and which window? Check: the same dates in GA4's Attribution paths report. Google says data-driven attribution can reattribute conversions for up to 7 days after the conversion, so the last week is provisional.
- Does the total match Shopify? Check: orders and revenue for the same dates and time zone. Expect a gap, and measure it once with your own query.
- Are orders counted once? Check: GA4 deduplicates purchases that share a transaction ID, so count distinct IDs and look for empty ones.
- Can I see the query? Check: run it in BigQuery and compare every number it quoted.
- What did you leave out? Check: refunds, test orders, empty values, direct visits. Count each exclusion yourself.
- Same question again: same numbers? Check: three runs in fresh chats. Providers document that outputs vary between runs.
- Which model, version and date? Check: a specific name, logged with the date. Providers retire models on published schedules.
- What would sales have been without this channel? Check: a pass says the export alone cannot tell, states its assumptions and names a test. One confident number is a story.
- What result would change your mind? Check: it names a holdout or geo test and what you would compare, such as total orders in test regions against control regions.
Which answers should stop you?
A fail on questions three to six means the numbers are not yours yet: reconcile before you read a channel split. A fail on seven or eight means you are comparing this week's answer with last week's from a different instrument. No export settles nine and ten; a holdout or geo test does. An LLM is a fair way to draft the query for question five and explain what comes back.
Where does a read fit?
Sources, 30 September 2026: Predicted Incrementality by Experimentation (PIE) for Ad Measurement (Gordon, Moakler and Zettelmeyer, arXiv, 2026); Close Enough? A Large-Scale Exploration of Non-Experimental Approaches to Advertising Measurement (Gordon, Moakler and Zettelmeyer, arXiv, 2022); Close Enough? journal record (Marketing Science, 2023, via RePEc); Get started with attribution (Google, 2026); Minimize duplicate key events with transaction IDs (Google, 2026); Advanced usage, reproducible outputs (OpenAI, 2026); Glossary, temperature (Anthropic, 2026); Model deprecations (Anthropic, 2026).
Related answers
Frequently asked questions
What is the first question to ask an attribution tool?
Which rule or method produced the number. GA4's Attribution reports offer three models, and all of them leave direct visits out of the credit unless the whole path is direct. A tool that cannot name its method cannot be checked.How do I check an LLM's attribution answer?
Ask for the query, run it in BigQuery, and compare orders and revenue with Shopify for the same dates and time zone. Ask the same question three times: the numbers should not change between runs.Is last-click attribution the same as incrementality?
No. In a preprint using 2,226 Meta ad experiments, 7-day last-click attribution gave an out-of-sample R-squared of 0.19 against incremental conversions per dollar. Incrementality comes from a holdout or geo test.
Go deeper: Incrementality testing, explained.
Sixty-second versions of these ideas: Causality Engine on YouTube Shorts.
Have an audience of ecommerce operators? Earn 20% recurring commission by recommending the €99 read.
Keep reading
Terms in this article
- AttributionAttribution identifies user actions that contribute to a desired outcome and assigns value to each. It reveals which marketing touchpoints drive conversions.
- Attribution ReportAttribution Report shows which touchpoints or channels receive credit for a conversion. It identifies which campaigns drive desired actions.
- ConversionConversion is a specific, desired action a user takes in response to a marketing message, such as a purchase or a sign-up.
- ExperimentationExperimentation in marketing conducts controlled tests to determine the causal impact of specific actions. This includes A/B testing and other controlled experiments to establish causality.
- ExperimentsExperiments are scientific procedures that test hypotheses or demonstrate facts. In marketing, experiments like A/B tests determine the causal effect of campaign changes, enabling data-driven decisions.
- Holdout TestA holdout test is an experiment that keeps a random part of an audience from seeing a campaign and compares its results with those of the part that could see it. The held-out part is the control group, so the gap in outcomes estimates what the campaign added.
- IncrementalityIncrementality is the part of an outcome, such as orders or revenue, that a marketing activity caused: the extra results compared with what would have happened without it. It is a causal effect, not the number of conversions credited to a channel.
- Incrementality TestingIncrementality Testing measures the additional impact of a marketing campaign. It compares exposed and control groups to determine causal effect.