What the Spider 2.0 SQL Benchmark Says About Marketing Data

Spider 2.0 includes GA4 in its tests: its paper says the databases come from industrial applications such as Google Analytics, and one printed example asks for sessions per traffic channel in a period.

What does Spider 2.0 test?

The paper (Lei et al., ICLR 2025 Oral, peer-reviewed) describes 632 real-world text-to-SQL workflow problems derived from enterprise-level database use cases, in systems such as BigQuery and Snowflake, often with over 1,000 columns. Its appendix describes two Google Analytics tasks: count the sessions for each traffic channel within a given time period, and compute the conversion rate from product-list clicks to product pages. The leaderboard splits the benchmark into three settings: Snow (547 tasks on Snowflake), Lite (547 tasks across BigQuery, Snowflake and SQLite) and DBT (68 project-level tasks). The page says the final output should be CSV files, not SQL, and it publishes gold answers for self-evaluation, so what is scored is the answer the query produces.

What do the scores show today?

The same benchmark gives different numbers by setting, by the system wrapped around the model and by date. Rows are submissions from the organisations listed beside them, and the page describes a submission process in which the maintainers validate methods. This is the top entry and a GPT-4o baseline for each setting, as read on 30 September 2026:

One model, three numbers, depending on where you read and when. The page itself says scores "may change slightly over time".

What does it leave out for attribution?

The two Google Analytics examples are descriptive: sessions per channel, a click-through rate. Each has a correct table to compare with. "What did this channel cause?" does not, because it asks about sales that did not happen, and data on the sales that did happen cannot answer it without a design or stated assumptions. Gordon, Zettelmeyer, Bhargava and Chapsky (Marketing Science, 2019, peer-reviewed) used 15 US advertising experiments at Facebook and found that observational methods often fail to reproduce the experimental effects. A holdout or geo test answers the causal question.

The databases are also not yours. Google says the BigQuery export can differ from the GA4 interface, and GA4 deduplicates purchases only when they carry a transaction ID. And a score belongs to one system: a row on this leaderboard is not the assistant you use.

How do you check SQL on your own data?

Take a closed month and do at small scale what the benchmark does: fix the answer first.

1. Write the plain query yourself: distinct transaction IDs and summed purchase revenue, by day.

SELECT
  event_date,
  COUNT(DISTINCT ecommerce.transaction_id) AS orders,
  SUM(ecommerce.purchase_revenue) AS revenue_as_exported
FROM `your_project.analytics_PROPERTY_ID.events_*`
WHERE _TABLE_SUFFIX BETWEEN '20260801' AND '20260831'
  AND event_name = 'purchase'
GROUP BY event_date
ORDER BY event_date;

1. Reconcile to Shopify. Compare orders for the same days and time zone. Expect a gap, and measure it once: that gap is your baseline, and a model-written query is judged against it, not against zero.
2. Ask the LLM the same question and for its SQL. Run that SQL yourself.
3. Pass: the same orders and revenue as your query on every day, or a difference it explains with a filter it stated, such as refunds. Fail: an unexplained difference, totals that match over days that do not, or a query that counts purchase events instead of distinct transaction IDs.
4. Repeat once with a filtered question, such as a single channel group.

An LLM is still a fair way to draft this query and explain its joins, and the leaderboard lists systems scoring far above the plain GPT-4o baseline. The check is what lets you use it without taking its numbers on trust.

Sources, 30 September 2026: Spider 2.0: Evaluating Language Models on Real-World Enterprise Text-to-SQL Workflows (Lei et al., arXiv, 2025); Spider 2.0 full text (Lei et al., arXiv, 2025); Spider 2.0 leaderboard (XLANG Lab, read 30 September 2026); A Comparison of Approaches to Advertising Measurement (Gordon et al., Marketing Science, 2019); BigQuery Export (Google, 2026); Minimize duplicate key events with transaction IDs (Google, 2026).

Related answers

• GPT-4 and Enterprise SQL: What the Benchmark Numbers Mean
• How to Test an LLM on Your Own Attribution Data
• Ten Questions to Ask Any Attribution Tool, LLM or Not
• Incrementality testing, explained
• Holdout test