Does your customer community cause retention?

Members of your customer community buy again more often than everyone else. That tells you who joins, not what joining does. To find out what the community causes, randomise the invitation, not the membership.

A recent funnel teardown ends with a brand's retention layer: email and SMS flows, a post-purchase upsell and a private Facebook group for customers (video). Private groups and communities are an easy tactic to copy and a hard one to measure, because the people who join choose to.

Why members always look better

Three reasons a member versus non-member comparison flatters the community:

• Keen buyers join. People who like the product enough to join a group were more likely to buy again anyway.
• Timing. People may join right after a good experience, such as an order that went well, when their next order was already more likely.
• Survival. Unhappy customers leave the group or never join, so the members you count are the ones who stayed.

All three are selection bias. The same problem sits under loyalty programmes, and how to measure loyalty programme ROI covers that version. It also flatters email subscribers, app users and anyone else who opted into something: the opting in is the tell.

If someone sells you a community, as a tool or as a service, the number they show you is likely to be member revenue, and member revenue is exactly the number that carries the selection.

The fix: randomise the invitation

You cannot randomly force people into a group, but you can randomly choose whom you invite.

The design:

1. Split new customers at random. For a fixed period, half of your new customers get the invitation in their post-purchase email and half do not. The uninvited half can still find the group on their own; you are only not asking.
2. Compare everyone, not just members. Over the same months, compare repeat purchase rate and revenue per customer between everyone who was invited and everyone who was not, whether or not they joined. That difference is the effect of the invitation.
3. Scale it to the joiners. Divide that difference by the difference in join rates between the two halves. The result estimates the effect of joining for the people who joined because they were invited.

The third step is the standard result for this kind of design, set out by Angrist, Imbens and Rubin in 1996: under a few clearly stated assumptions, the ratio is the average effect for the people whose joining depended on the invitation. It is not the effect for your most devoted members, who would have joined anyway, and that is the honest limit of the design.

It works because the coin flip, not the customer, decided who was asked. That is the logic of a randomised controlled trial, applied to the one part of the community you control.

How to tell with the data you already have

If you cannot run the test yet, two checks narrow the question.

1. Before and after, against a match. For members, compare purchase frequency in the months before they joined with the months after, next to non-members matched on first order date and first order value. Treat the result as likely to overstate the effect, since people may join after a good experience.
2. Look for accidents. If some customers never received the invitation because of a template change or a sending error, compare them with those who did, as long as the reason they missed it had nothing to do with who they are.

Read both by cohort of first order month, the way which channels bring repeat buyers reads channels, so an old loyal cohort does not stand in for a new one.

What to measure in each half

Readings for the invited and the uninvited, over the same months:

• Repeat purchase rate at fixed points after the first order, chosen before the test starts.
• Revenue per customer, counting customers with no second order as zero rather than leaving them out.
• Discount use, since offers posted in a group can pull orders forward at a lower margin.
• Unsubscribes and complaints, since an invitation is one more message in a busy inbox.

Compare the halves at every reading, never members with non-members. The test needs enough new customers in each half to show the difference that would change your decision, so size it first; how long an incrementality test should run covers the sizing, and the same thinking decides how long the payback clock in CAC payback period for ecommerce should run before you judge it.

What to test next

Once the invitation test has a reading, the same design answers the next questions.

• Timing of the invitation. At the first order, or after the second.
• What happens inside the group. Product education or offers. Offers posted in a community can move orders that an email would have moved anyway.
• Who gets asked. Whether an invitation brings back customers who have not ordered in a while is a different question from whether it keeps new ones. Invite a random half of lapsed customers and compare the halves the same way.
• Hygiene. Randomise at the customer level, and keep the two halves apart in your email tool so nobody receives both messages.

Rule of thumb: if you cannot say who would have been in the group anyway, you do not yet know what the group does. The comparison you want is the one in incremental sales, not correlations, and a holdout test is how you get it.

Related answers

• How to measure loyalty programme ROI
• Which channels bring repeat buyers, not just first orders
• CAC payback period for ecommerce
• Incremental sales, not correlations
• Products with more reviews sell more. So what?