250 listing changes, zero measurable effect
An agent can rewrite every listing in minutes. Simultaneous change destroys the contrast that any attribution method needs in order to separate one effect from another.
Three platforms, three answers, one budget to defend. Your GA4 export already holds the answer, and the first finding is free and never leaves your browser. Get your first finding free
By Joris van Huët, Founder & CEOUpdated 2 min read
The speed that makes an agent worth using is the same property that makes its work unmeasurable. Not a flaw in the agent. A property of simultaneous change.
Why simultaneity is the problem
Every method for working out what caused a result needs contrast: some part of the world where the thing happened and some part where it did not, or a before and an after with nothing else moving.
An agent that rewrites 250 listings, regenerates ad copy and selects new creative in one afternoon removes all of it. Revenue moves. Nothing in the data says which change moved it.
What you are left with
| Evidence you have | What it supports |
|---|---|
| Revenue before and after the batch | That something changed, in aggregate |
| Per-product revenue after | Nothing causal, since all products changed |
| Ad platform reporting | The platform's own view, which already overcounts |
That is a thin basis for the next decision, which is usually "do more of what worked." You cannot do more of what worked if you cannot name it.
This is not an argument against agents
Speed is genuinely valuable, and a human doing 250 listings over three weeks introduces its own confound, because three weeks of everything else also happened.
The argument is for spending a small part of the speed on structure. An agent that can change 250 listings can just as easily change 200 and leave 50 alone.
The cheapest structure that works
Hold some back. A set of products left untouched through the batch, chosen before you look at results, gives you the contrast the method needs. It costs you the upside on those products for one window.
That is a holdout, and it is the only thing on this list that turns an observation into evidence.
What a read gives you afterwards
Without a holdout the same read will tell you honestly that the design does not support a causal claim, which is worth knowing before you scale.
Related answers
Frequently asked questions
Why can't I measure what an AI agent changed?
Because it changed everything at once. Causal methods need contrast: something changed here and not there, or before and after with nothing else moving. A simultaneous batch removes that contrast entirely.Is it better to make changes slowly by hand?
Not necessarily. Three weeks of manual changes means three weeks of everything else also changing, which is its own confound. The fix is structure, not slowness: hold part of the catalogue back.What is the cheapest way to keep agent work measurable?
Leave a set of products untouched through the batch, chosen before you see any results. That holdout costs the upside on those products for one window and is what turns an observation into evidence.
Go deeper: Causal attribution, explained.
Sixty-second versions of these ideas: Causality Engine on YouTube Shorts.
Have an audience of ecommerce operators? Earn 20% recurring commission by recommending the €99 read.
Keep reading
Terms in this article
- AttributionAttribution identifies user actions that contribute to a desired outcome and assigns value to each. It reveals which marketing touchpoints drive conversions.
- IncrementalityIncrementality is the part of an outcome, such as orders or revenue, that a marketing activity caused: the extra results compared with what would have happened without it. It is a causal effect, not the number of conversions credited to a channel.
- RevenueRevenue is the total income generated by the sale of goods or services related to a company's primary operations.
- ROASROAS: Return on Ad Spend measures the revenue generated for each dollar spent on advertising. Causal attribution calculates an accurate ROAS by showing the true impact of ad campaigns.