Natural Language to SQL for Marketing: The Gap Between Demo and Reality

A natural-language question fixes less than it seems to. "Revenue by channel last month" leaves open which revenue, which channel, which day and which currency, and the generated SQL picks one of each without saying so. Google's developers say GA4's session-level channel attribution is "neither directly available in BigQuery export nor can it be calculated with full accuracy", so a channel query on the export is an approximation before any model writes a line. The Spider 2.0 leaderboard has climbed, and a January 2026 audit found annotation errors in 76 of the 121 Spider 2.0-Snow examples whose gold SQL is public, so a score is a weak guide to your data.

The leaderboard has climbed far past the figures it still quotes in its own introduction. Five sources say why a score is still not the number you need.

• The leaderboard, read 30 September 2026. The Spider 2.0 leaderboard (XLANG Lab) lists a top score of 96.70 on its Snowflake table (547 examples) and 76.23 on its mixed BigQuery, Snowflake and SQLite table (547 examples). The top rows are agent systems submitted by named teams. The page's introduction still quotes 17.1% for o1-preview and 10.1% for GPT-4o. It says the scores "may change slightly over time", and that it has released "all examples and gold answers for self-evaluation", so the questions are public.
• An audit of that benchmark. Jin et al. (preprint, January 2026) reviewed the 121 Snowflake examples whose gold SQL is public and found annotation errors in 76 of them, "an error rate of 62.8%". The benchmark team's own update cut the rate only from 66.1% to 62.8%. The audit covers only those 121 of 547 examples, and no source cited here measures how the errors affect the current leaderboard scores.
• Private data. BEAVER (preprint, revised May 2026) built its questions from private data warehouses and reports that agentic systems on GPT-5.2 "achieve only 10.8% accuracy".
• Definitions. A paired test (preprint, April 2026) gave three frontier LLMs 100 questions over a retail dataset, first with the schema alone and then with a 4 KB hand-written document of "measures, conventions, and disambiguation rules". The document "improves accuracy by +17 to +23 percentage points across all three models", lifting scores from 45.5-50.5% to 67.7-68.7%.
• Repeat runs. Ouyang et al. (arXiv paper, revised October 2024) asked ChatGPT for code on 829 problems across three benchmarks and report that the share of tasks with "zero equal test output across different requests" was 75.76%, 51.00% and 47.56%, and that "setting the temperature to 0 does not guarantee determinism". It covers 2023-era code generation, not SQL.

None of these measures how often an LLM's revenue-by-channel query matches a Shopify store's own reports. The scores describe systems on fixed questions, and the definitions test covers one retail dataset.

What does "revenue by channel last month" leave open?

1. Which revenue. Shopify documents a total price "before returns" and another "after returns" for the same order, and GA4 has its own purchase revenue.
2. Which channel. Google's developer guidance says the export holds attribution "at user (first visit) and event level", not GA4's session-level model.
3. Which day. The property's reporting time zone, UTC or the store's. See the time zone post.
4. Which currency. Local, reporting or USD, covered in the same post.
5. Which orders. Test orders, cancelled orders and repeated purchase IDs. See the duplicates post.

How do I check an answer I can't read?

Write your definitions once and paste them above every question:

Definitions for every question:
- Revenue: ecommerce.purchase_revenue on purchase events, one row per transaction_id.
- Day: the GA4 property's reporting time zone (event_date).
- Currency: one currency per total; never add across currencies.
- Channel: collected source and medium on the purchase event, grouped as <your mapping>.
- Exclude test orders. List any assumption you add.

Then run four steps:

1. Ask for the SQL and the assumptions. Run the SQL yourself in a dry run first, then on the real table.
2. Ask the same question in two fresh chats and diff the two queries. A difference is either a choice your question left open or run-to-run variation. Pin it down in the definitions either way.
3. Reconcile. The channel rows must add up to your total orders and revenue for the same dates and time zone, within gaps you can list order by order.
4. Pass or fail. Pass: same definitions, same totals, an explained gap. Fail: queries that differ between chats, or channel rows that do not add up.

An LLM can be a good interface for this: it drafts the query and explains it in plain words. The number that moves budget should come from a method you can check.

Where does a causal read fit?

Sources, 30 September 2026: Spider 2.0 leaderboard (XLANG Lab, read 30 September 2026); Pervasive Annotation Errors Break Text-to-SQL Benchmarks and Leaderboards (Jin et al., arXiv, 2026); BEAVER (arXiv preprint, 2026); Semantic Layers for Reliable LLM-Powered Data Analytics (arXiv preprint, 2026); Bridge the gap between the Google Analytics UI and BigQuery export (Google for Developers, 2023); Shopify Admin API, Order (Shopify, 2026).

Related answers

• Schema size and what a model is shown
• What the same SQL means in each warehouse
• The Spider2-SQL benchmark, read carefully
• Where LLMs help in marketing
• Causal attribution