LLM Hallucinations in Marketing Data: When Your AI Invents Revenue

An LLM can give you a revenue figure that appears nowhere in your data, in the same tone it uses for a correct one. The fastest check on your own numbers is to recompute the total from your export.

How often do models state what their source does not support?

Vectara's leaderboard gave each model the same private set of over 7700 articles, asked for summaries "using only the facts presented in the document", and scored them with Vectara's own evaluation model, HHEM-2.3. The best-listed model still has a rate above zero. This is a vendor-published measurement of summaries of articles, not of revenue tables, and the board is updated regularly, so read the current version before you quote it.

DSBench (Jing et al., arXiv preprint, 2024, revised 2025) tested models and agents on 466 data analysis tasks sourced from Eloquence and Kaggle competitions.

For the mechanism, Kalai and colleagues (arXiv preprint, 2025) argue that language models "hallucinate because the training and evaluation procedures reward guessing over acknowledging uncertainty". A peer-reviewed survey (Huang et al., ACM Transactions on Information Systems) describes the outcome as "plausible yet nonfactual content".

None of these sources tested marketing exports, and this page cites no measurement that did. There is no rate here for your case, only a reason to check.

What counts as an invented number?

Three failures get called hallucination, and each needs its own check:

1. Absent. The figure is not in your data: revenue for a channel with no rows in the export, or for a campaign that never ran.
2. Misread. The data is there and the arithmetic is wrong: a filter that drops refunds, an order counted twice, two currencies added together.
3. Imported. A typical value from the model's training text, such as a "normal" conversion rate, presented as if it came from your store.

How do you check a number before it reaches a budget?

Run these on any figure an LLM gives you from your export:

1. Start small. Paste ten orders whose total you can add up by hand and ask for revenue by channel. Include one channel that has no orders.
2. Recompute the real total. Pivot the raw export for the same dates and filters, or use SUMIFS. Then compare it with Shopify orders and revenue, and GA4 purchases, for the same period and time zone.
3. Ask for the evidence. Request the query or formula and the number of rows it used, then run it yourself. Ask for the order IDs behind the largest figure and look three of them up in Shopify.
4. Write down what you ran. Model name, date, prompt and file, so you can repeat the check next month.

Pass: your total matches to the cent, or the gap has a cause you can name (refunds, tax, shipping, a time-zone cut-off), and the model says plainly that the channel with no orders has no data. Fail: a figure you cannot reproduce from your rows, or a number returned for data that does not exist.

An LLM still earns its place on the parts you can check: it can draft the query or formula and explain a finished table in plain words. The number stays yours to recompute.

Sources, 30 September 2026: Hallucination Leaderboard (Vectara, updated 22 September 2026, vendor-published); DSBench: How Far Are Data Science Agents from Becoming Data Science Experts? (Jing et al., arXiv, 2024); Why Language Models Hallucinate (Kalai et al., arXiv, 2025); A Survey on Hallucination in Large Language Models (Huang et al., ACM Transactions on Information Systems).

Related answers

• How to test an LLM's confidence against your own numbers
• Why LLM totals go wrong: SUM, AVG and COUNT
• Audit trails for LLM attribution answers
• Causal attribution for ecommerce brands