Does an LLM Leave an Audit Trail for Attribution Numbers?

Not by default: the trail that holds up is the one you write. Researchers who ran five LLMs configured to be deterministic saw accuracy variations up to 15% across runs, and Anthropic's API reference says that even with temperature 0.0 "the results will not be fully deterministic". So log the prompt, the pinned model ID, the date, the SQL and the result, and keep the result.

What does the evidence say about repeatability?

Three sources: one arXiv preprint, and two providers' own documentation (vendor statements, not independent measurements).

• Same model, same settings. Atil et al. (arXiv preprint, revised 2025) ran five LLMs configured to be deterministic on eight common tasks, 10 runs each, zero-shot and few-shot. They report "accuracy variations up to 15% across naturally occurring runs with a gap of best possible performance to worst possible performance up to 70%", and that "none of the LLMs consistently delivers repeatable accuracy across all tasks". The tasks were general, not SQL on marketing data.
• Same model ID. Anthropic's documentation says each model ID "identifies a pinned version of the model", but that "infrastructure updates produce minor differences in observable behavior even when the model ID and weights have not changed", and that an alias is "a convenience pointer that resolves to the most recent dated snapshot".
• Different snapshot. OpenAI's guide says "Even different snapshots of models within the same family could produce different results" and recommends "Pinning your production applications to specific model snapshots".

No public measurement was found of how often LLM-written attribution queries return different totals on a repeat run. The reproduction test below measures it on your data.

What should each log row hold?

Eight fields:

• Question and prompt, verbatim, with any definitions you pasted.
• Model ID as called: a pinned snapshot, not an alias.
• Run time, in UTC.
• Data source and version: the table or export file. Google's BigQuery export help says to query events_YYYYMMDD rather than events_intraday_YYYYMMDD, "so you're querying a stable dataset for the day".
• The SQL or formula, as run.
• The result, stored in a named table or file, with the control total it reconciles to.
• Who ran it and who reviewed it.
• Definition choices: revenue column, time zone, currency.

What do you get without building anything?

If your data sits in BigQuery, it already logs every query job. The JOBS view "displays running jobs along with job history for the past 180 days", with the SQL text, the creation time, the user or service account and whether the results came from a cache. It holds nothing about the prompt or the model unless your tool sets job labels, so keep those in your own log.

-- replace region-eu with the region of your data
SELECT creation_time, user_email, job_id, query, labels, cache_hit, total_bytes_billed
FROM `region-eu`.INFORMATION_SCHEMA.JOBS
WHERE creation_time > TIMESTAMP_SUB(CURRENT_TIMESTAMP(), INTERVAL 7 DAY)
  AND statement_type = 'SELECT'
ORDER BY creation_time DESC;

How do you reproduce a number?

1. Pick one logged number from last month.
2. Re-run the stored SQL against the same data. BigQuery's time travel "covers the past seven days by default"; beyond that, use the stored export file or result. A same-day re-run can be a cache hit, so check the cache_hit column above.
3. Compare with the stored result. Pass: identical to the cent, or the difference traced to a named cause such as a late refund. Fail: a difference nobody can explain.
4. Ask the model the same question with the pinned model ID, and compare the SQL, not the prose. A different query isn't a fail. A different total from a query you can't justify is. An LLM is fine for drafting the SQL; the log is what lets someone else check it.

-- the data as it stood at the logged time, inside the time travel window
SELECT SUM(revenue) AS total_revenue
FROM `project.dataset.orders`
FOR SYSTEM_TIME AS OF TIMESTAMP('YYYY-MM-DD HH:MM:SS+00');

Where does a causal read fit?

Later, once the log exists, a causal attribution read like Causality Engine's gives each number a receipt: every figure checks against your GA4 export, recomputed in your browser.

Sources, 30 September 2026: Non-Determinism of "Deterministic" LLM Settings (Atil et al., arXiv, 2025); Create a Message (Anthropic, 2026); Model IDs and versioning (Anthropic, 2026); Text generation (OpenAI, 2026); BigQuery Export (Google, 2026); JOBS view (Google Cloud, 2026); Data retention with time travel (Google Cloud, 2026).

Related answers

• Would you present LLM-generated attribution numbers to your board?
• Your data analyst or an LLM: who owns attribution?
• The reproducibility problem with LLM analytics
• LLM attribution and compliance: GDPR, CCPA and your data
• Attribution audit