Products with more reviews sell more. So what?

Products with more reviews sell more. Part of that is the arrow running backwards: they have more reviews because they sold more. Before you spend on collecting reviews, or copy the proof on a competitor's best page, test what the proof itself does.

Social proof runs through several recent ecommerce videos. A resale-marketplace guide lists reviews among its tips for more sales (video). A dropshipping tutorial treats a product's public sold count as evidence of demand and does the revenue maths on screen (video). A funnel teardown files testimonial ads as an angle of their own (video). All three treat proof as a cause of sales. Sometimes it is. Your data cannot tell you how often unless you design it to.

Why the obvious comparison fails

Look at your catalogue: the products with the most reviews are probably your best sellers. That is not evidence that reviews sell. Reviews accumulate from buyers, so a product that sells well collects them faster.

Three things hiding in a reviews-versus-sales chart:

• Reverse causality. Sales produce reviews, so the chart would slope upwards even if reviews persuaded nobody.
• Selection. Better products earn both more sales and better reviews. The product did it, not the stars. That is selection bias in its plainest form.
• Your own budget. Products you promote get more visitors, more buyers and so more reviews. Your spending decisions are drawn into the chart.

The same trap sits under a bestseller is not evidence of why. Copying a competitor's review-heavy page adds the survivorship problem from copying bestsellers: you only see the pages that worked.

How researchers got around it

Judith Chevalier and Dina Mayzlin compared the same books' sales on two retailers' sites, each with its own reviews (Journal of Marketing Research). Comparing one book on two sites holds the book's own appeal fixed: its author, cover, subject and season affect both sites alike, and the reviews are what differ. They found that an improvement in a book's reviews on one site raised its sales there relative to the other site, and that 1-star reviews moved sales more than 5-star reviews did (NBER working paper).

The lesson for a store is the design rather than their effect size: hold the product fixed and change only the proof. That is the counterfactual a catalogue-wide chart never gives you.

How to tell what proof does on your store

Two free checks come first.

1. Within product, over time. For products that went from no reviews to a few, compare their conversion rate in the weeks before and after the first reviews appeared, next to similar products whose reviews did not change in the same weeks. It is rough, but it holds the product fixed.
2. Next to your returns. Set review content beside return reasons for the same products. Reviews that promise something the product does not deliver can lift orders and returns together, so read orders net of returns.

What to test

Three tests, cheapest first:

1. Hide the proof for a random half. If your review app or theme allows it, hide star ratings on collection pages for a random half of visitors, or on a random half of matched products, for a fixed period. Compare conversion and orders net of returns.
2. Test a sold counter the same way. A public sold count or a popular badge is a claim about other shoppers. Show it to a random half of visitors. If it only lifts products that were already selling, it is decorating winners rather than creating sales.
3. Test proof in the ad, not only on the page. Hold the product, offer and format fixed and change only whether the ad carries a customer quote. Run it where you control who sees which version, because an ad platform's own split shows each version to different people, as explained in your Meta creative test is not an A/B test.

Reading the result:

• Per product, not only in total. Proof may matter more where shoppers know least, such as a new product or a higher price, and less on a product they already recognise. The test shows whether that holds for your catalogue, which tells you where collecting reviews is worth the effort.
• On orders net of returns. A badge that lifts orders and returns together has moved less than it seems.
• At a length you fixed in advance. A split halves the traffic each version sees, so decide the length before you start, as in how long an incrementality test should run, and do not stop the moment a difference appears.

A review app is sold on collecting more reviews, so any report from it that compares reviewed and unreviewed products starts on the side of the arrow that flatters it.

Only test proof that is real. Invented reviews and inflated counters are not a test variable, and a store caught with them has a bigger problem than its conversion rate.

Where this leaves the product page

Proof probably helps. The test tells you how much, on which products and for which visitors, and whether the budget you would spend collecting more of it would do better elsewhere. The same logic runs through what a winning product page is hiding: the visible feature of a winner is not always the reason it won.

Related answers

• A bestseller is not evidence of why
• Copying bestsellers is survivorship bias
• What a winning product page is hiding
• Vary one element, learn something reusable
• Does your customer community cause retention?