250 listing changes, zero measurable effect: An agent can rewrite every listing in minutes. Simultaneous change destroys the contrast that any attribution method needs in order to separate one effect from another.
Read the full article below for detailed insights and actionable strategies.
Key insight
How much Meta over-reports ROAS after iOS privacy changes
The speed that makes an agent worth using is the same property that makes its work unmeasurable. Not a flaw in the agent. A property of simultaneous change.
Why simultaneity is the problem
Every method for working out what caused a result needs contrast: some part of the world where the thing happened and some part where it did not, or a before and an after with nothing else moving.
An agent that rewrites 250 listings, regenerates ad copy and selects new creative in one afternoon removes all of it. Revenue moves. Nothing in the data says which change moved it.
What you are left with
| Evidence you have | What it supports |
|---|---|
| Revenue before and after the batch | That something changed, in aggregate |
| Per-product revenue after | Nothing causal, since all products changed |
| Ad platform reporting | The platform's own view, which already overcounts |
That is a thin basis for the next decision, which is usually "do more of what worked." You cannot do more of what worked if you cannot name it.
This is not an argument against agents
Speed is genuinely valuable, and a human doing 250 listings over three weeks introduces its own confound, because three weeks of everything else also happened.
The argument is for spending a small part of the speed on structure. An agent that can change 250 listings can just as easily change 200 and leave 50 alone.
The cheapest structure that works
Hold some back. A set of products left untouched through the batch, chosen before you look at results, gives you the contrast the method needs. It costs you the upside on those products for one window.
That is a holdout, and it is the only thing on this list that turns an observation into evidence.
What a read gives you afterwards
With a holdout in place, a causal read on a Google Analytics export returns an estimate with a confidence interval rather than a before-and-after difference. It is 99 euro once, refunded if it does not move a budget decision.
Without a holdout the same read will tell you honestly that the design does not support a causal claim, which is worth knowing before you scale.
The interactive demo shows the fields it uses, with no signup.
Related answers
Key Terms in This Article
Analytics
Analytics is the systematic computational analysis of data. It reveals customer behavior and measures campaign performance.
Attribution
Attribution identifies user actions that contribute to a desired outcome and assigns value to each. It reveals which marketing touchpoints drive conversions.
Causality
Causality is the relationship where one event directly causes another, essential for identifying specific actions that drive desired outcomes in marketing.
Confidence Interval
Confidence Interval is a statistical range of values that likely contains the true value of a metric. In marketing analytics, it quantifies uncertainty around estimates, indicating the precision of an outcome or causal effect.
Google Analytics
Google Analytics is a web analytics service that tracks and reports website traffic.
Incrementality
Incrementality measures the true causal impact of a marketing campaign. It quantifies the additional conversions or revenue directly from that activity.
Revenue
Revenue is the total income generated by the sale of goods or services related to a company's primary operations.
Related Articles
Sixty-second versions of these ideas: Causality Engine on YouTube Shorts.
Ready to see your real numbers?
Own the budget? Upload your GA4 export and see which channels drive incremental sales, with confidence intervals, in minutes. Have to defend it? Start with the live demo and take the read to your CFO.
Full refund if you don't see value.
Stay ahead of the attribution curve
Weekly insights on marketing attribution, incrementality testing, and data-driven growth. Written for the person who owns the budget and the person who has to defend it.
No spam. Unsubscribe anytime. We respect your data.
Frequently Asked Questions
Why can't I measure what an AI agent changed?
Because it changed everything at once. Causal methods need contrast: something changed here and not there, or before and after with nothing else moving. A simultaneous batch removes that contrast entirely.
Is it better to make changes slowly by hand?
Not necessarily. Three weeks of manual changes means three weeks of everything else also changing, which is its own confound. The fix is structure, not slowness: hold part of the catalogue back.
What is the cheapest way to keep agent work measurable?
Leave a set of products untouched through the batch, chosen before you see any results. That holdout costs the upside on those products for one window and is what turns an observation into evidence.