Why an LLM Gives Different Answers to the Same Data Question

Because output varies between runs even at settings meant to be deterministic, and your data moves underneath it. OpenAI's documentation calls chat completions non-deterministic by default, and Anthropic's says that even at temperature 0 results will not be fully deterministic. Freeze the query and the data, and judge a tool on its numbers, not its wording.

What has been measured?

• Run to run. Atil et al. (arXiv, submitted 6 August 2024, revised 2 April 2025) evaluated five LLMs configured to be deterministic on eight common tasks across 10 runs.
• Code, the nearest thing to SQL. Ouyang, Zhang, Harman and Wang (arXiv, revised 17 October 2024) studied ChatGPT on 829 code generation problems from three benchmarks.
• Documentation and data. OpenAI says the seed parameter gives "mostly" deterministic output, and Anthropic's page for Claude Sonnet 5 says setting temperature, top-p or top-k to non-default values returns a 400 error. On the data side, Google says GA4 processing can take 24 to 48 hours and credit for a key event can change for up to 12 days.

All three papers are arXiv preprints; the rest is official documentation.

How do you check your own tool?

1. Freeze the data. Export a closed period ending more than 12 days back to a dated CSV or table, so the data cannot move.
2. Ask the same question at least three times, in fresh chats. Log the tool, the model name it shows, the date, the exact prompt, the numbers returned and any SQL.
3. Compare the numbers. Pass: identical in every run, whatever the wording. Fail: any numeric difference. You have measured the tool's own variation.
4. Run the saved SQL twice on the frozen table. The two results should match each other and the numbers the tool quoted. If they do not, the tool quoted numbers its own query does not produce, and because the query is text you can see why.
5. Keep the audit trail. The prompt, model string, date, SQL and file name together. Rerun when the model behind the tool changes.

Is a repeatable answer a correct one?

No. A tool can repeat a wrong number every time. Repeatability is the floor; correctness comes from reconciling to Shopify and GA4, and for causal claims from a holdout or geo test. An LLM is still a good way to draft SQL and explain it: once the SQL is saved, the repeatable part is the query, not the model.

Later, if you want numbers you can recompute, a causal attribution read like Causality Engine's lets you check every number against the GA4 export, recomputed in your browser.

Sources, 30 September 2026: Non-Determinism of "Deterministic" LLM Settings (Atil et al., arXiv, 2025); An Empirical Study of the Non-determinism of ChatGPT in Code Generation (Ouyang et al., arXiv, 2024); How is ChatGPT's behavior changing over time? (Chen, Zaharia and Zou, arXiv, 2023); Advanced usage, reproducible outputs (OpenAI, 2026); Glossary, temperature (Anthropic, 2026); Claude Sonnet 5 model page (Anthropic, 2026); Data freshness (Google, 2026).

Related answers

• How to Test an LLM on Your Own Attribution Data
• Ten Questions to Ask Any Attribution Tool, LLM or Not
• GPT-4 and Enterprise SQL: What the Benchmark Numbers Mean
• Causal attribution, explained