Would You Present LLM-Generated Attribution Numbers to Your Board?
Only if someone else can reproduce each number and the slide says whether it was credited or measured. Give each figure a query, a run date, a model ID and a reconciliation to Shopify before the meeting.
By Joris van Huët, Founder & CEOUpdated 4 min read
Only if someone else can reproduce each number, and the slide says whether it measures what the spend caused or only what was credited. The second half carries the risk: in 15 US advertising experiments at Facebook, observational methods "often fail to produce the same effects as the randomized experiments". A specific to check today: GA4's purchase revenue is net of refunds, and a Shopify orders export keeps Subtotal, Shipping, Taxes and Total in separate columns, so a slide that doesn't say which one it used can't be reconciled.
What should a board member ask about an attribution number?
Four questions to answer before the meeting:
- Where does it come from? Name the export or table and the definition: which revenue column, which time zone, which currency.
- Does it tie to the books? The total for the same dates should match Shopify orders, with any gap explained in euros.
- Would someone else get the same number? That needs the query, the run date and, if an LLM wrote the query, the model ID.
- Did the spend cause it, and how sure is that? Tracked conversions show what was credited. What the spend caused needs a holdout, a geo test or assumptions you state on the slide.
What does the evidence say?
Four measured results, each with its limits.
Credited is not caused (peer-reviewed). Gordon, Zettelmeyer, Bhargava and Chapsky (Marketing Science, 2019) compared randomized results from 15 US Facebook experiments, with 500 million user-experiment observations and 1.6 billion ad impressions, against multiple observational models. Their conclusion: "The observational methods often fail to produce the same effects as the randomized experiments, even after conditioning on extensive demographic and behavioral variables."
Even a randomized test is noisy (peer-reviewed). Lewis and Rao (Quarterly Journal of Economics, 2015) report on 25 field experiments with major US retailers and brokerages: "The median confidence interval on return on investment is over 100 percentage points wide." A measured number should carry its range on the slide.
Writing the SQL is scored on public leaderboards. The BIRD leaderboard reports execution accuracy. In its oracle-knowledge table, human performance (data engineers and database students) is listed at 92.96 on the test set, and the best submitted system, entered 26 September 2026, at 82.95. The Spider 2.0 leaderboard, a separate benchmark of 547 text-to-SQL tasks on Snowflake, lists a top score of 96.70. Neither tests whether a figure matches your finance team's definition of revenue.
The same question can get different answers (preprint). Atil and colleagues (arXiv preprint, revised 2025) ran five LLMs configured to be deterministic on eight common tasks, 10 runs each, and report "accuracy variations up to 15% across naturally occurring runs". Their tasks weren't SQL on marketing data. No public measurement was found of how often LLM-written attribution figures tie to a store's own orders, so the check below measures it on yours.
What is the check to run before the meeting?
Build a provenance table for the three numbers on the slide:
- One row per number: the definition, the source export or table, the query or formula, the run date, the model ID, and the reconciliation total.
- Reconcile: sum the same revenue column over the same dates in a fresh Shopify orders export and compare it with the slide. Write the gap in euros and name each cause: refunds, time zone, currency or duplicate orders.
- Re-run: have a second person run the stored query from scratch. If an LLM wrote it, run the stored SQL, not a new prompt.
- Label: mark each number "credited" (attribution) or "measured" (holdout or geo test).
Pass: the re-run matches and every gap has a named cause. Fail: a gap that closes only when someone edits a filter, or a number nobody can re-run. An LLM is useful for drafting the query and formatting the table. The reconciliation only counts if it runs against the raw Shopify export.
Where does a causal read fit?
Later, once the table exists, a causal attribution read like Causality Engine's can add what each channel caused, next to last-click, from the same GA4 export.
Sources, 30 September 2026: A Comparison of Approaches to Advertising Measurement (Kellogg School of Management, Marketing Science 2019); The Unfavorable Economics of Measuring the Returns to Advertising (Lewis and Rao, Quarterly Journal of Economics, 2015); BIRD leaderboard (BIRD-bench, 2026); Spider 2.0 leaderboard (XLANG Lab, 2026); Non-Determinism of "Deterministic" LLM Settings (Atil et al., arXiv, 2025); Analytics dimensions and metrics (Google, 2026); Export orders (Shopify, 2026).
Related answers
Frequently asked questions
Can I show AI-generated attribution numbers to my board?
Yes, if each number carries its query, run date, model ID and a reconciliation to Shopify orders, and is labelled credited or measured. A number nobody can re-run is a claim, not a figure.What should I log when an LLM writes the attribution query?
The prompt, the exact model ID called, the date, the SQL produced and the result.How do I reconcile an LLM's revenue figure to Shopify?
Sum the same revenue column over the same dates in a fresh Shopify orders export. Subtotal is before shipping and taxes; Total is the total cost of the order. Explain any gap in euros by cause: refunds, time zone, currency or duplicates.
Go deeper: Incrementality testing, explained.
Sixty-second versions of these ideas: Causality Engine on YouTube Shorts.
Have an audience of ecommerce operators? Earn 20% recurring commission by recommending the €99 read.
Keep reading
Terms in this article
- Ad ImpressionAd Impression is a single instance of an advertisement displaying on a webpage. Impressions are a key input for models measuring the causal impact of ad exposure on user behavior.
- AttributionAttribution identifies user actions that contribute to a desired outcome and assigns value to each. It reveals which marketing touchpoints drive conversions.
- Causal AttributionCausal Attribution uses causal inference to determine which marketing touchpoints genuinely cause conversions, not just correlate with them.
- Confidence IntervalConfidence Interval is a statistical range of values that likely contains the true value of a metric. In marketing analytics, it quantifies uncertainty around estimates, indicating the precision of an outcome or causal effect.
- ConversionConversion is a specific, desired action a user takes in response to a marketing message, such as a purchase or a sign-up.
- ExperimentsExperiments are scientific procedures that test hypotheses or demonstrate facts. In marketing, experiments like A/B tests determine the causal effect of campaign changes, enabling data-driven decisions.
- IncrementalityIncrementality is the part of an outcome, such as orders or revenue, that a marketing activity caused: the extra results compared with what would have happened without it. It is a causal effect, not the number of conversions credited to a channel.
- Incrementality TestingIncrementality Testing measures the additional impact of a marketing campaign. It compares exposed and control groups to determine causal effect.