Your Data Analyst vs. an LLM: Who Should Own Attribution?

A named person should own it, and that person can use an LLM. On BIRD's test set the best listed system scores 82.95 against a human reference of 92.96, while the top listed score on one Spider 2.0 set is 96.70. Either way, writing the query is only part of the job. The rest is the definitions, the reconciliation to Shopify and the sign-off, and the benchmarks below don't score those.

What do benchmarks say about LLMs as analysts?

Four measured results (leaderboards and peer-reviewed conference papers), all dated and all limited:

• Text-to-SQL, BIRD. The leaderboard reports execution accuracy. In its oracle-knowledge table, human performance (data engineers and database students) is listed at 92.96 on the test set, and the best submitted system, entered 26 September 2026, at 82.95.
• Text-to-SQL, Spider 2.0. The paper (Lei et al., ICLR 2025) reports that an o1-preview-based code agent "successfully solves only 21.3% of the tasks" in its 632-task benchmark. The project's leaderboard now lists 96.70 at the top of a separate 547-task Snowflake set. Benchmarks move, and different task sets aren't comparable.
• Statistical and causal reasoning with data, QRData. Liu et al. (Findings of ACL 2024) posed 411 questions with data sheets: "The strongest model GPT-4 achieves an accuracy of 58%", and "models encounter difficulties in data analysis and causal reasoning".
• Open-ended analysis, BLADE. Gu et al. (EMNLP 2024) built 12 datasets with ground truth from independent analyses by expert data scientists, and report that language models are "often limited to basic analyses".

No public measurement was found of LLM accuracy against analysts on marketing attribution questions. The QRData result dates from 2024, so it says nothing about later models.

What does the owner of attribution own?

Three things, in writing:

• Definitions. What counts as an order (refunds, cancellations, test orders), which revenue column, which time zone and currency, and how Direct is treated. GA4's purchase revenue is net of refunds, and a Shopify orders export keeps Subtotal, Shipping, Taxes and Total apart, so someone has to choose.
• Reconciliation. The totals tie to Shopify for the same dates, with each gap explained.
• Sign-off. A name next to the number and a record of what was run.

Where an LLM helps is drafting a query from those definitions, explaining a join someone else wrote and summarising a finished table. It isn't the person the board can ask next week why the number moved.

How do you compare an analyst and an LLM on your own data?

1. Pick a question with a known answer, such as net revenue by channel for one closed week.
2. Give both the same definitions in writing: revenue column, time zone, currency, refunds.
3. Ask both for the number and the query, and reconcile each to a fresh Shopify orders export and to GA4 for the same dates.
4. Diff the two queries, not just the totals.

Pass: each can name every difference from Shopify's total, and the queries agree on the filters. Fail: a total that matches only after a filter changed, or a column nobody defined. Keep the question and its control total: it is the test the next model version has to pass.

Sources, 30 September 2026: BIRD leaderboard (BIRD-bench, 2026); Spider 2.0 leaderboard (XLANG Lab, 2026); Spider 2.0: Evaluating Language Models on Real-World Enterprise Text-to-SQL Workflows (Lei et al., ICLR 2025); Are LLMs Capable of Data-based Statistical and Causal Reasoning? (Liu et al., Findings of ACL 2024); BLADE: Benchmarking Language Model Agents for Data-Driven Science (Gu et al., EMNLP 2024); Analytics dimensions and metrics (Google, 2026); Export orders (Shopify, 2026).

Related answers

• Does an LLM leave an audit trail for attribution numbers?
• Would you present LLM-generated attribution numbers to your board?
• The Spider benchmark and your marketing data
• What LLMs are useful for in marketing analytics
• Attribution audit