Which of the agent's changes moved ROAS?
When several changes shipped together, some questions remain answerable and others do not. Telling which is which, before spending a week digging into the wrong one.
Three platforms, three answers, one budget to defend. Your GA4 export already holds the answer, and the first finding is free and never leaves your browser. Get your first finding free
By Joris van Huët, Founder & CEOUpdated 2 min read
Run the numbers for your store: the free break-even ROAS calculator.
Some of what you want to know is recoverable after the fact. Most is not.
Recoverable: anything that varied across products
If the agent changed descriptions on some products and not others, even accidentally, that variation is a natural experiment. It is weaker than a designed one, because the selection was not random, but it is real contrast and it can be read.
Check the batch diff before concluding everything changed. Agents frequently skip items for mundane reasons, and the skipped set is a gift.
Recoverable: anything with a staggered rollout
Changes that reached different products or markets on different days give you timing variation. If the stagger was driven by processing order rather than by product quality, it is close to random and that is usable.
Not recoverable: levers that moved together everywhere
If descriptions, titles and images all changed on every product on the same afternoon, no analysis separates them. Not with more data, not with a better model, not with a longer window.
The honest answer is that the question cannot be answered with this data, and the useful response is to design the next batch so it can be.
Not recoverable: anything coinciding with an external event
A batch shipped the same week as a promotion, a press mention or a platform algorithm change is entangled with it permanently.
The triage table
| Question | Answerable? |
|---|---|
| Did the batch as a whole move revenue | Weakly, before and after |
| Did descriptions specifically move it | Only if some products kept the old ones |
| Did the change work better in one category | Yes, if categories were treated differently |
| Which of four simultaneous levers mattered | No |
What to do with the unanswerable ones
Record them as unanswered rather than estimating them. A number invented to fill a gap in a report gets quoted later as though it were measured, and that is worse than an admitted blank.
Every channel gets a data-health score from 0 to 100 that grades the read, not the channel, and a next step: raise, cut or hold, or fix tagging first.
Related answers
Frequently asked questions
Can I work out which agent change mattered after the fact?
Sometimes. Anything that varied across products or rolled out in stages gives real contrast that can be read. Levers that moved together everywhere on the same day cannot be separated by any analysis.Why check the batch diff first?
Because agents skip items for mundane reasons, and the skipped set is accidental contrast. Assuming everything changed can throw away the only comparison group you have.What should I do about questions the data cannot answer?
Record them as unanswered. A number invented to fill a gap gets quoted later as though it had been measured, which is worse than an admitted blank.
Go deeper: Causal attribution, explained.
Sixty-second versions of these ideas: Causality Engine on YouTube Shorts.
Have an audience of ecommerce operators? Earn 20% recurring commission by recommending the €99 read.
Keep reading
Terms in this article
- AttributionAttribution identifies user actions that contribute to a desired outcome and assigns value to each. It reveals which marketing touchpoints drive conversions.
- Attribution ReportAttribution Report shows which touchpoints or channels receive credit for a conversion. It identifies which campaigns drive desired actions.
- Natural ExperimentNatural Experiment is an empirical study where experimental and control conditions are determined by nature or external factors. This estimates causal effects when randomization is not feasible.
- Product PageProduct Page is a webpage dedicated to a single product. It includes images, descriptions, pricing, and purchase options.
- RevenueRevenue is the total income generated by the sale of goods or services related to a company's primary operations.
- TaggingTagging labels or mentions another user, page, or location in a social media post. It creates links between entities, which helps analyze influence and attribution.