On-site search and recommendations: how to measure the lift

Hold a search or recommendation block back from a random share of visitors and compare orders per visitor, because clicks on the block overstate what it adds. In a randomised field experiment at an online book retailer, personalised recommendations raised shoppers' propensity to buy by 12.4% and basket value by 1.7% (Li, Grahl and Hinz, 2022). In one site's browsing logs for 2.1 million users, at least 75% of the activity from recommendation click-throughs would likely have happened without the recommendations (Sharma, Hofman and Watts, 2015). A widget report counts what passed through the block. Only a holdout shows what the block added.

What does a widget report count?

Shopify's Search & Discovery reports give recommendations a purchase rate: the number of customers who purchased a product that they discovered in recommendations (Shopify Help Center, official documentation). The search reports do the same for search results. That is attribution: the order passed through the block. It says nothing about whether the shopper would have bought anyway, and the observational estimate above puts most recommendation click-throughs in that group.

The same trap sits in any comparison of shoppers who used search with shoppers who didn't. In Baymard's usability testing, roughly half of participants turned to search as their preferred way to find products, and the rest used the main navigation (Baymard Institute, 2026). People choose to search, so that comparison sets two kinds of shopper against each other, not search against no search.

What do randomised tests show about recommendations and search?

Three results from peer-reviewed field experiments, each at one retailer or platform, so read the direction and not the size:

1. They can add orders. In the book-retailer experiment above, random assignment to recommendations raised the chance of buying and basket value, so the effect showed up in orders and not only in clicks.
2. They move sales between products. On a fashion retailer's site, conditional on a product's page view, its own sales decreased by 1.9% and the sales of its recommended substitutes increased by 9%. On average, a product's recommendation links produced an 11% gain in total sales of the product and its recommended substitutes (Kumar and Hosanagar, 2019). Count orders across the set, not one product's sales.
3. They trade use with search. In a randomised experiment with 555,800 customers on a large ecommerce platform, less relevant home-page recommendations significantly increased use of the search channel, which the authors read as a partial substitution effect (Yuan and colleagues, 2025). Change one block and the other moves, so read total orders. For search on its own, no source read here reports a randomised test of a search change on orders, so treat search as the case you test, not one the literature settles.

How do you hold a block back on your own store?

Run it on your own traffic:

1. Pick one block: the recommendation row on product pages, or the search suggestions. One change per test.
2. Assign visitors, not page views. Give each visitor a group on the first visit and keep it, so nobody sees both versions.
3. Label the group in GA4 before launch. Send it as an event parameter and register it as a custom dimension, because Google says reporting on a custom dimension starts 24 to 48 hours after you create it (Google Analytics Help).
4. Judge on orders per visitor and revenue per visitor for everyone in each group over the same days. Keep clicks on the block as a diagnostic only.
5. Check the split before you read the result. If you planned an even split and the groups differ by more than chance allows, something assigned visitors unevenly. Microsoft researchers warn that ignoring a sample ratio mismatch can make a bad change look good, or the reverse (Fabijan and colleagues, 2019).
6. Run whole weeks and fix the end date first. The holdout test planner turns your daily orders, the smallest lift worth detecting and the share held out into a number of days.

What the result means:

• Pass: orders per visitor are higher where the block shows, by more than chance allows at your traffic, and average order value and refunds are not worse.
• Fail: clicks on the block rise and orders per visitor stay level. The block rerouted shoppers and added no orders.
• Too small to read: the gap sits inside the noise. Extend the test, or keep the answer as a direction and say so.

Sources, 30 September 2026: How Do Recommender Systems Lead to Consumer Purchases? A Causal Mediation Analysis of a Field Experiment (Information Systems Research, 2022); Estimating the Causal Impact of Recommendation Systems from Observational Data (ACM Conference on Economics and Computation, 2015); Measuring the Value of Recommendation Links on Product Demand (Information Systems Research, 2019); How Recommendation Affects Customer Search: A Field Experiment (Information Systems Research, 2025); Shopify Search & Discovery reports and analytics (Shopify Help Center, 2026); Ecommerce Search UX 2026: 8 Search Query Types UX Best Practices (Baymard Institute, updated 2026); About custom dimensions and metrics (Google Analytics Help, 2026); Diagnosing Sample Ratio Mismatch in Online Controlled Experiments (Microsoft Research, 2019). The sizes above come from single retailers and one platform; the experiments are randomised, the browsing-log estimate is observational.

Related answers

• Site search in GA4: what the search events tell you
• Merchandising changes: measure the page, not the ad
• A/B testing: sample size first
• Holdout test, defined
• Incrementality testing