An AI agent found a cheaper supplier. Now measure it

A supplier switch has a price you see on day one and a cost you see weeks later. An AI sourcing agent is built to win the first number. The second one shows up in delivery times, returns, reviews and repeat orders, and by the time it arrives it is easy to blame your ads for it.

A recap of Alibaba.com's CoCreate 2026 event is doing well on ecommerce YouTube this week. It demos AI agents that find suppliers, check them and negotiate prices, and it contrasts that with the months of manual outreach sourcing used to take (the video). Faster sourcing means more switches. Each switch changes the product your ads, emails and search listings are selling, and it deserves to be measured as the change it is.

Where the cost shows up

A worse supplier shows up in roughly this order:

1. Delivery time, if the new supplier ships slower or from further away.
2. Returns and refund reasons: defects, sizing, and items not as described.
3. Reviews, a few weeks after delivery.
4. Repeat purchase: customers who would have come back do not.
5. Channel mix: the channels that live on returning customers, such as email, direct and branded search, are where a worse product shows first in your marketing numbers.

The fifth line is where the misreading starts. When returning-customer revenue softens a month after the switch, the usual suspects are creative fatigue, the ad platform and the email calendar. The switch is easy to forget, because it happened a month ago and it saved money. That is a confounding problem in its plainest form: two changes, one visible suspect.

Who reports the saving

The agent that negotiated the price reports the price. That is its job, and the saving is real. It has no view of your returns desk and no reason to wait for one before it reports success. A sourcing tool is judged on the deals it finds, not on how the product holds up six weeks later. The incentive to measure the rest belongs to you alone.

How to tell, from your own data

Date the switch, then compare cohorts at a fixed horizon:

1. Record the switch date for each product as a GA4 annotation (Analytics Help) and in your own changelog.
2. Split orders by supplier: before the date and after it, for the switched products.
3. Compare the same five numbers for each group: delivery days, return rate, refund reasons, review rating and repeat purchase rate, at a horizon you choose before you look. If you already run cohort analysis, most of this is a filter away.
4. Add a comparison group: products you did not switch, over the same weeks. If both groups moved, the cause is something they share, such as the season, your carrier or your ads. If only the switched products moved, the supplier is the likely cause.

That last step, switched products against unswitched ones over the same period, is a difference in differences read. It is the cheapest honest check you have, and it needs nothing but your own order data.

A clean result is useful either way. If nothing moved on the switched products relative to the rest, the agent found you a real saving, and you now hold a dated record that says so the next time someone asks why margins changed. If returns rose only on the switched products, you caught it in one product line instead of across the whole catalog.

What to test before you commit

• Run both suppliers side by side for a period: alternate batches, or ship one region from each. Then compare the two groups of orders on the same numbers.
• Write down the decision rule first: which metrics, which horizon, and what result sends you back to the old supplier.
• Keep the listing, price and ads unchanged during the test, or you will not know which change did what.
• Switch one product line at a time. An agent that changes ten suppliers in a week leaves you nothing to compare against. The changelog is the asset, and a change you cannot undo is a test you cannot run.

If your agent can act on its own, give it the same limits you would give a new hire: one change at a time, a record of every change, and a holdout it is not allowed to touch. Designing a holdout an agent will respect covers how.

What the margin math should include

The saving the agent reports is unit cost times units. The number you care about is margin after everything the switch touched:

• Unit cost saving, the part you already know.
• Minus refunds and return shipping on the extra returns, if there are any.
• Minus the repeat orders you lose, valued at their margin, over your horizon.
• Minus any support time spent on complaints about the new batch.

If what remains is still positive, the switch paid. If you cannot fill in the lines yet, the horizon has not passed, and the switch is still a test, whatever the dashboard says. The same thinking sits behind returns-adjusted ROAS: a sale is not finished until the return window has closed.

Related answers

• The changelog is the asset, not the output
• The cost of a change you cannot undo
• Designing a holdout an agent will respect
• Returns-Adjusted ROAS: The Metric Your Dashboard Is Hiding
• How to Set Up Cohort Analysis on Shopify