Which of the agent's changes moved ROAS?: When several changes shipped together, some questions remain answerable and others do not. Telling which is which, before spending a week digging into the wrong one.
Read the full article below for detailed insights and actionable strategies.
The numbers behind the problem
Articles analyzed
Glossary terms
Platform integrations
Starting price
Some of what you want to know is recoverable after the fact. Most is not. Knowing which is which saves a week.
Recoverable: anything that varied across products
If the agent changed descriptions on some products and not others, even accidentally, that variation is a natural experiment. It is weaker than a designed one, because the selection was not random, but it is real contrast and it can be read.
Check the batch diff before concluding everything changed. Agents frequently skip items for mundane reasons, and the skipped set is a gift.
Recoverable: anything with a staggered rollout
Changes that reached different products or markets on different days give you timing variation. If the stagger was driven by processing order rather than by product quality, it is close to random and that is usable.
Not recoverable: levers that moved together everywhere
If descriptions, titles and images all changed on every product on the same afternoon, no analysis separates them. Not with more data, not with a better model, not with a longer window.
The honest answer is that the question cannot be answered with this data, and the useful response is to design the next batch so it can be.
Not recoverable: anything coinciding with an external event
A batch shipped the same week as a promotion, a press mention or a platform algorithm change is entangled with it permanently.
The triage table
| Question | Answerable? |
|---|---|
| Did the batch as a whole move revenue | Weakly, before and after |
| Did descriptions specifically move it | Only if some products kept the old ones |
| Did the change work better in one category | Yes, if categories were treated differently |
| Which of four simultaneous levers mattered | No |
What to do with the unanswerable ones
Record them as unanswered rather than estimating them. A number invented to fill a gap in a report gets quoted later as though it were measured, and that is worse than an admitted blank.
A causal read on a Google Analytics export applies exactly that discipline: an estimate with a confidence interval where the design supports one, an explicit label where it does not. 99 euro once, refunded if it does not move a budget decision. The interactive demo shows both outcomes on sample data, with no signup.
Related answers
Key Terms in This Article
Analytics
Analytics is the systematic computational analysis of data. It reveals customer behavior and measures campaign performance.
Attribution
Attribution identifies user actions that contribute to a desired outcome and assigns value to each. It reveals which marketing touchpoints drive conversions.
Attribution Report
Attribution Report shows which touchpoints or channels receive credit for a conversion. It identifies which campaigns drive desired actions.
Causality
Causality is the relationship where one event directly causes another, essential for identifying specific actions that drive desired outcomes in marketing.
Confidence Interval
Confidence Interval is a statistical range of values that likely contains the true value of a metric. In marketing analytics, it quantifies uncertainty around estimates, indicating the precision of an outcome or causal effect.
Google Analytics
Google Analytics is a web analytics service that tracks and reports website traffic.
Natural Experiment
Natural Experiment is an empirical study where experimental and control conditions are determined by nature or external factors. This estimates causal effects when randomization is not feasible.
Product Page
Product Page is a webpage dedicated to a single product. It includes images, descriptions, pricing, and purchase options.
Related Articles
Sixty-second versions of these ideas: Causality Engine on YouTube Shorts.
Ready to see your real numbers?
Own the budget? Upload your GA4 export and see which channels drive incremental sales, with confidence intervals, in minutes. Have to defend it? Start with the live demo and take the read to your CFO.
Full refund if you don't see value.
Stay ahead of the attribution curve
Weekly insights on marketing attribution, incrementality testing, and data-driven growth. Written for the person who owns the budget and the person who has to defend it.
No spam. Unsubscribe anytime. We respect your data.
Frequently Asked Questions
Can I work out which agent change mattered after the fact?
Sometimes. Anything that varied across products or rolled out in stages gives real contrast that can be read. Levers that moved together everywhere on the same day cannot be separated by any analysis.
Why check the batch diff first?
Because agents skip items for mundane reasons, and the skipped set is accidental contrast. Assuming everything changed can throw away the only comparison group you have.
What should I do about questions the data cannot answer?
Record them as unanswered. A number invented to fill a gap gets quoted later as though it had been measured, which is worse than an admitted blank.