Machine-translated store pages: test what they cost

If you sell in more than one language, a clumsy translation is a trust signal on every page it touches. Comparing conversion rates between countries will not tell you what it costs. A test inside one market will.

A recent comedy video turns spotting sketchy online stores into a game (video). Broken translations are one of the running jokes, next to AI product photos and scammy design. For a brand selling across borders, that joke is a measurement question: are your machine-translated pages costing you orders, and how would you know?

Why the country comparison misleads

The obvious check is to compare conversion rate by country or language and blame the translation when a market lags. It will mislead you, because almost everything else differs between markets too.

What differs between two markets besides the translation:

• Delivery times and costs, and the returns route.
• The payment methods shoppers expect.
• Currency, price points and local competitors.
• How well known the brand is, and which channels send the traffic.
• Seasons, holidays and paydays.

Any of these can explain a gap, which makes each of them a confounding variable for the translation question. A market that converts worse and has machine translation may convert worse because of its delivery costs.

The comparison can also hide a cost. A market with strong demand can convert well in spite of a poor translation, which makes the translation look harmless there when it is still losing you orders you never see.

How to tell whether the translation is the problem

Start with free checks on your own data.

1. Compare languages inside one country. GA4 records the language of the shopper's browser or device in its Language and Language code dimensions, and the country separately (Google's dimension reference). In a country where you serve more than one language version, compare shoppers on each. Delivery, payment and prices are the same, so the comparison is closer to fair, though the language groups can still differ in who they are.
2. Find where each version loses people. Compare the steps from product page to cart, cart to checkout and checkout to purchase for each language version. If the translation is the problem, the losses should sit where words carry the decision: product pages and checkout. A delivery or payment problem would tend to show at checkout.
3. Read what shoppers ask. Look at support questions and return reasons by language. Questions about things the page already answers in your home language suggest the translation did not answer them.

These are signals. The proof is a test.

What to test

Three designs, strongest first:

1. Split the visitors. If your platform can serve two versions of a page to randomly split visitors, show the edited and the machine versions of the same pages to different halves of the traffic in one market. Same market, same products, same weeks: the translation is the only difference.
2. Split the pages. In one market, have a fluent speaker rewrite half of your product pages, chosen at random from matched pairs with similar price, category and traffic, and leave the other half machine translated. Compare conversion rate and return reasons between the halves over the same weeks.
3. Step and compare. If neither split is possible, rewrite one market's pages on a known date and compare its conversion before and after with a similar market that did not change, over the same weeks. It is the weakest of the three and still far better than a league table of countries.

Read every result on orders net of returns, not on add-to-cart. A page that persuades by being unclear can win the order and lose it again at delivery, when the product is not what the shopper understood. Returns-adjusted ROAS covers the return side.

A translation app sells coverage and speed, and what a clumsy sentence costs at checkout is not in its report. Nobody but you will measure it.

What to measure, beyond conversion rate

Five readings per version, over the same weeks:

1. Orders per visitor, the headline number.
2. Checkout completion, where delivery terms, returns text and payment instructions carry the weight.
3. Return rate and return reasons, where a misunderstood description shows up weeks after the order.
4. Support contacts per order, where confused shoppers ask what the page failed to tell them.
5. Average order value, in case clearer pages change what people buy as well as whether they buy.

A split halves the traffic each version gets, so run it on the pages with enough orders to read, and fix the length before you start rather than stopping when the numbers look good. How long an incrementality test should run covers the sizing. Start with the pages that carry the most revenue and with checkout, where one unclear sentence sits in front of every order.

The same test works for any sitewide copy change

The design carries over to AI-rewritten descriptions, new size guides or a new tone of voice. Change half of the matched pages, keep the other half, and read orders net of returns. It follows the rule in vary one element, learn something reusable: one change, one reading.

What it avoids is the sitewide switch on a single date, which an AI rebuilt page breaks channel attribution shows is so hard to read afterwards. A change that lands everywhere at once leaves you nothing to compare it with.

Related answers

• Vary one element, learn something reusable
• An AI rebuilt page breaks channel attribution
• Returns-adjusted ROAS
• What a winning product page is hiding
• The trust check that hands your sale to search