Prompt Engineering for Attribution: The Myth of the Perfect Question

There is no perfect question: a better specification raises the odds of a right analysis without making it right. Ask your own question five ways on one export and see which numbers survive.

Can a better prompt fix an attribution answer?

Evidence (preprint, September 2026): CausalVerify, from Madrid academics, scores language models' causal analyses end to end across difference-in-differences, event-study, instrumental-variable and regression-discontinuity designs. Joint recovery means getting both the effect estimate and its standard error right, and a seed is one simulated dataset. The authors conclude that "a plausible method and runnable code do not guarantee recovery".

What it does not show: marketing data, or a prompt test. The design swap is the nearest thing to a better prompt that this post cites, and it is not a prompt test. On the two newest follow-up pairs, the gain was confined to models from the family that built the references.

Is the SQL problem already solved?

Largely on one setting, and less so on the others. Evidence (benchmark and leaderboard): Spider 2.0, an ICLR 2025 benchmark of 632 real-world text-to-SQL problems, reports that its code agent on o1-preview solved 21.3% of them (Lei et al., 2025).

What it does not show: the benchmark scores results against gold answers, and "what did paid social cause?" has none in an orders export. That is reasoning from how the benchmark is built, not a finding.

Does the wording of a question change the verdict?

Evidence (peer-reviewed, PNAS 2026; Amazon AWS and Carnegie Mellon authors): across three datasets, independent AI analysts given the same hypothesis showed "substantial dispersion in effect sizes", p-values and conclusions, and reassigning the persona or the model "shifts the distribution of results even among methodologically sound runs" (Bertran, Fogliato and Wu). OpenAI's guide adds that even different snapshots of one model family "could produce different results."

Those datasets were a soccer-referee study, a coding trial and an election survey, not marketing exports. No source cited here measures how far prompt wording moves an attribution split on marketing data.

Does "control for seasonality" make the answer causal?

Not by itself. Evidence (peer-reviewed journal, 2019): in 15 Facebook ad experiments, "the observational methods often fail to produce the same effects as the randomized experiments, even after conditioning on extensive demographic and behavioral variables" (Gordon et al., Marketing Science, Kellogg and Facebook authors). That paper tested statistical methods, not prompts; a prompt asking for more conditioning asks for the same adjustment.

Opinion: treat a prompt as a hypothesis about what your question means, and a holdout as the answer key. An assistant that writes both the query and the verdict grades its own homework.

How do you test a prompt on your own data?

Run this on your own export:

1. Fix the inputs. Pick one date range. Export Shopify orders and GA4's channel report for it, in one time zone and currency.
2. Ask five ways. In five fresh chats, ask for revenue by channel: last touch, first touch, "what drove", "what caused", and once with "control for seasonality". Save the query or calculation behind each answer.
3. Reconcile. Each answer's total orders and revenue should equal Shopify's for that range. Pass: equal to the order. Fail: any gap, meaning rows were dropped or counted twice.
4. Compare the splits. Pass: the splits differ only where the stated rule differs, such as the window or the credit rule, and each answer names that rule. Fail: the split moves with the wording and no answer says why.
5. Hold something out. Switch one channel off in some regions for long enough to count orders, then compare total orders with the regions where it stayed on. Keep the answer that agrees with the holdout. If none does, the prompt was not the problem.

Prompts earn their place in translation: drafting the SQL, then explaining a result you have already reconciled.

Sources, 30 September 2026: CausalVerify (Zhang et al., arXiv, 2026); Spider 2.0 paper (Lei et al., arXiv, 2025); Spider 2.0 leaderboard (XLANG Lab, 2026); Many AI Analysts, One Dataset (Bertran et al., PNAS, 2026; arXiv version); OpenAI text generation guide (OpenAI, 2026); A Comparison of Approaches to Advertising Measurement (Gordon et al., Marketing Science, 2019).

Related answers

• Same question, different answer: why LLM analytics is hard to reproduce
• What the Spider SQL benchmark says about marketing data
• Where LLMs help in marketing
• Incrementality testing
• Holdout test