How to Test an LLM on Your Own Attribution Data
Write questions whose answers you can compute from your own GA4 and Shopify exports, ask the LLM each one three times, and grade its results against your answers. Add two causal questions and grade them on whether the tool admits the export cannot settle them.
By Joris van Huët, Founder & CEOUpdated 4 min read
Grade it against answers you computed yourself, and grade the causal questions separately. Whether it gets yours right is a grading job, and an export alone does not settle a causal question.
What do published tests show?
- Descriptive questions. AD-Bench (Hu et al., submitted 15 February 2026, revised 22 June 2026) is built from real marketing-analysis requests on a production advertising platform. Its best model, Claude-Opus-4.7, attains Pass@1 = 76.9% and Pass@3 = 80.4% overall, and drops to Pass@1 = 61.4% on the hardest level.
- Causal questions. CausalDS (Leban and Sun, arXiv preprint, 9 July 2026) evaluates six models on generated data. All six recover structure essentially perfectly, while models differ in uncertainty quantification and abstention, which the benchmark scores as a first-class outcome. The data is synthetic.
- Real advertising effects. Gordon, Zettelmeyer, Bhargava and Chapsky (Marketing Science, 2019, peer-reviewed) used 15 US advertising experiments at Facebook comprising 500 million user-experiment observations. The observational methods often fail to produce the same effects as the randomized experiments.
No public measurement exists of how often an LLM's answer to "which channel caused my sales?" matches a holdout result on a real store's data. Until one does, your own export is the evidence.
How do you test an LLM on your own data?
The Spider 2.0 benchmark page publishes gold answers for self-evaluation. Do the same at store scale:
- Freeze the data. Export a closed period from GA4 and the same dates from Shopify, ending at least 12 days back: Google says attribution credit for a key event can change for up to 12 days.
- Drop personal columns. The European Commission lists names, home addresses, email addresses, IP addresses and cookie IDs as examples of personal data; this test needs none of them.
- Write the answer key first. Eight questions you can answer in a spreadsheet or SQL: orders, revenue, revenue by channel group, refunds, new against returning buyers. Write the numbers down.
- Ask each question three times, in fresh chats. OpenAI's documentation calls chat completions non-deterministic by default, and Anthropic's says results at temperature 0 are not fully deterministic. Log the tool, the model name, the date, the prompt and each answer.
- Grade results, not prose. Counts must match exactly; fix your tolerance for money before you start. Run any query the tool quotes: a different number from the one it quoted is a fail.
- Add two causal questions. "Which channel caused the most sales?" and "What happens to revenue if I cut one channel's spend?" A pass says the export alone cannot settle it, states its assumptions or asks for a holdout or geo test. A fail is one confident number with no design.
Pass: every arithmetic question matches your key on all three runs, and the causal answers state their limits. Fail: any miss, any difference between runs, or a causal answer with no design behind it.
What does a pass tell you?
Less than it feels like: this tool, this model version, this export, this date. Providers retire models. Anthropic says it regularly retires older ones, and its page for a model released on June 30, 2026 already marks it legacy. Rerun the set when the model behind the tool changes.
An LLM is still a fair way to draft a query and explain a column. The test is what lets you use it without taking its numbers on trust.
Sources, 30 September 2026: AD-Bench (Hu et al., arXiv, 2026); CausalDS (Leban and Sun, arXiv, 2026); A Comparison of Approaches to Advertising Measurement (Gordon et al., Marketing Science, 2019); Spider 2.0 leaderboard (XLANG Lab, read 30 September 2026); Data freshness (Google, 2026); Data protection explained (European Commission, 2026); Advanced usage, reproducible outputs (OpenAI, 2026); Glossary, temperature (Anthropic, 2026); Model deprecations (Anthropic, 2026); Claude Sonnet 5 model page (Anthropic, 2026).
Related answers
Frequently asked questions
How many questions do I need to test an LLM on my data?
Ten is enough for a first read: eight you can answer by arithmetic and two causal ones. Write the answers before you ask, run each question three times, and grade results, not wording. Add a question whenever a real one fails.Can I paste my Shopify orders into ChatGPT?
Remove personal columns first. The European Commission lists names, home addresses, email addresses, IP addresses and cookie IDs as examples of personal data, and this test needs only order IDs, dates, channels and amounts.Why run each question three times?
Providers document run-to-run variation: OpenAI calls chat completions non-deterministic by default, and Anthropic says results at temperature 0 are not fully deterministic. A number that changes between runs fails before you compare it with your key.
Go deeper: Incrementality testing, explained.
Sixty-second versions of these ideas: Causality Engine on YouTube Shorts.
Have an audience of ecommerce operators? Earn 20% recurring commission by recommending the €99 read.
Keep reading
Terms in this article
- AnalyticsAnalytics is the systematic computational analysis of data. It reveals customer behavior and measures campaign performance.
- AttributionAttribution identifies user actions that contribute to a desired outcome and assigns value to each. It reveals which marketing touchpoints drive conversions.
- CounterfactualA counterfactual is what would have happened under conditions that did not occur, such as a customer's outcome had they not received the campaign they did receive. It is never observed, so every effect estimate has to stand in for it with a control group, comparable units or a forecast.
- ExperimentsExperiments are scientific procedures that test hypotheses or demonstrate facts. In marketing, experiments like A/B tests determine the causal effect of campaign changes, enabling data-driven decisions.
- IncrementalityIncrementality is the part of an outcome, such as orders or revenue, that a marketing activity caused: the extra results compared with what would have happened without it. It is a causal effect, not the number of conversions credited to a channel.
- Incrementality TestingIncrementality Testing measures the additional impact of a marketing campaign. It compares exposed and control groups to determine causal effect.
- RevenueRevenue is the total income generated by the sale of goods or services related to a company's primary operations.
- ShopifyShopify is an ecommerce platform for creating online stores and selling products. Attribution modeling shows which marketing channels drive traffic and conversions within Shopify.