Skip to content

Control Group

A control group is the part of an experiment that does not get the treatment being tested, so its outcomes stand in for what the treated group would have had without it. In marketing it is the audience, or the set of regions, held back from a campaign.

By , Founder & CEOUpdated 6 min read

Run the numbers for your store: the free holdout test planner.

What is Control Group?

In a comparative experiment, the control group gets the standard treatment, a placebo or nothing, as Wikipedia describes it. The reason to have one is a missing-data problem. For every treated unit, the outcome it would have had untreated is never observed. Cunningham's textbook, Causal Inference: The Remix, calls this the fundamental problem of causal inference and notes that no amount of data removes it. A control group supplies an estimate of the missing outcome from other units.

Whether the estimate is any good depends on how the group was formed. Wikipedia notes that, for conclusions to be valid, the units in the treatment and control groups must be representative of the same population, and that random assignment from a common population is one way to achieve it. Under random assignment the groups have the same potential outcomes on average, though one finite sample still differs by chance. A group formed by people's own choices is different. Cunningham says that when the individual user chooses their own treatment, selection bias should be the first thought, and in his account of an eBay experiment nearly all of an observational estimate of the return on branded search ads was selection bias.

In marketing, control groups are built in a few ways. Meta's help center defines its control group as the audience not eligible to see the ads, while its test group is everyone eligible, including people who never see one. Google describes users held back from seeing ads in the control group. For regions, Google Research describes geo experiments in which regions that do not overlap are assigned at random to control or treatment. A control group that gets exposed anyway weakens the comparison: Google says contamination of the control group lowers the estimated difference between treatment and control.

For illustration, suppose a store emails 9,000 of its 10,000 subscribers and holds out a random 1,000. If 450 of the 9,000 order, against 40 of the 1,000 held out, the order rates are 450 / 9,000 = 5 percent and 40 / 1,000 = 4 percent. The control group suggests that about 4 percent would have ordered anyway, which is 0.04 x 9,000 = 360 of the 450 orders, leaving 450 - 360 = 90 orders that the email added.

Why Control Group matters for ecommerce

A count of orders after a campaign cannot separate the campaign from everything else that changed. The IAB and IAB Europe guidelines on incremental measurement list seasonality, pricing, promotions, competitor activity and unmeasured concurrent campaigns as factors that can make results misleading, and say attribution and ROAS describe what happened and cannot say whether marketing caused it. A control group is the piece that turns an order count into a comparison. The gap between a naive comparison and an experiment can be large. At eBay, field experiments found no measurable short-term benefit from brand-keyword ads, and paid search as a whole returned only a fraction of what conventional non-experimental estimates implied.

How to use Control Group

  1. Name the treatment (one campaign, channel or email flow) and the outcome (orders or revenue per customer) before you look at any results.
  2. Decide what you can withhold the treatment from: customers on a list you control, an audience inside an ad platform's lift tool, or regions.
  3. Assign units by chance before launch. In a spreadsheet, give each unit a random number, put those below your cut-off in the control group, and save the list with the date.
  4. Check balance: average a pre-launch measure, such as past orders per customer, in each group. Cunningham notes that you can check observed variables yourself by averaging them separately for treatment and control. A gap far larger than chance would produce is a sign the assignment did not work.
  5. Keep the control group out of the treatment, including through other channels, devices and regions, and measure the same outcome the same way in both groups over the same window.
  6. Check the result. Pass: the groups looked alike before launch, the control stayed unexposed, and the gap is larger than the noise expected from the group sizes. Fail: the groups differed before launch, the control was reached, or the gap is within the noise. Then treat the answer as unknown, not as no effect.

Common mistakes

  1. Treating people who did not see or click the ad as the control group. They differ from the people who did before the campaign starts, so the gap mixes the effect of the campaign with who they are. In Cunningham's eBay example, nearly all of the observational estimate was selection bias.
  2. Comparing with a different period, such as last month or the same weeks last year. Seasonality, price changes and promotions move outcomes at the same time, and IAB flags them among the factors that can make results misleading.
  3. Letting the control group be exposed through another channel or platform. IAB names control group exposure on an unmeasured platform as a source of contamination, and Google says contamination reduces the reported incrementality of a campaign.
  4. Making the control group too small to detect a realistic effect. A Quarterly Journal of Economics paper by Lewis and Rao reports that individual-level sales are very volatile, so a small group gives a noisy answer, and a noisy null result is easy to misread as no effect.
  5. Applying the result to every campaign. A control group shows what this campaign added on top of everything else running in this window; Google says a lift study calculates the incremental value of the measured campaigns on top of any other active marketing.

Frequently asked questions

  • What is a control group in marketing?
    It is the audience, customer list or set of regions held back from a campaign so that its results can be compared with those of the group that received it. Its outcomes estimate what the treated group would have done without the campaign, which holds only if assignment was random or otherwise unrelated to how people would have bought anyway.
  • Is an unexposed audience the same as a control group?
    No. People who did not see or click an ad were not assigned by chance, so they usually differ from people who did. A control group is formed by random assignment before the campaign, and Meta's design compares everyone eligible to see the ads, including people who never saw one, with a control group that was not eligible.
  • Can you build a control group without randomizing?
    You can build a comparison group by matching or modeling, but then equal groups rest on assumptions instead of a random draw. A 2019 Marketing Science paper on a set of Facebook advertising experiments found that observational estimates often differed from the randomized results even after adjusting for extensive demographic and behavioral data.
  • What is the difference between a control group and a holdout group?
    In marketing the two usually mean the same group. Meta's help center describes the holdout group as the part of the ad audience deliberately kept from seeing the ads, and notes that it is also called the control group. A holdout test is the experiment built around that group.

Further reading

Your platforms guess.
We run the math.

Upload a GA4 export and see what each channel caused, next to last-click, in 1–2 minutes. The read is yours to keep.

Get your first finding free

Free, in your browser: your file is not uploaded. The full read is €99, refundable within 30 days. Prices exclude VAT.
Or book a 30-min call.