Counterfactual

What is Counterfactual?

The word has a philosophical use and a statistical one. In philosophy, a counterfactual is a conditional about what would have been the case had something been otherwise, and counterfactual theories of causation explain what causal claims mean with conditionals such as 'if c had not happened, e would not have happened'. That analysis is contested: the Stanford Encyclopedia of Philosophy reports that decades of debate have left doubts about any simple counterfactual analysis of singular causation. In statistics the same idea appears as potential outcomes. Each unit has an outcome under each treatment, only the one it actually received is observed, and the other is the counterfactual.

A treatment's effect is defined by comparing the two. Cunningham's textbook describes a causal effect as a comparison of two states of the world, the outcome actually observed under one intervention against the counterfactual outcome under the other. One customer cannot both receive and not receive a campaign, so the effect for a single customer cannot be calculated. The textbook calls this the fundamental problem of causal inference, cites Holland (1986) for it, and says a bigger dataset cannot fix it.

So every estimate uses a stand-in. A randomized control group stands in for the treated group's counterfactual because random assignment balances observed and unobserved variables across the groups, at least in large samples. Without randomization, a comparison group or a forecast from the treated unit's own history has to do the job, with assumptions that need defending. In a structural causal model, a counterfactual is computed by first updating the probabilities with what was observed and then replacing the equation for the changed variable with its hypothetical value. A 2015 paper on measuring marketing interventions builds the counterfactual market response with a state-space model and discusses its strengths and limitations where a randomized experiment is unavailable.

For illustration, suppose a store emails 1,000 randomly chosen customers and holds out another 1,000 randomly chosen customers. Suppose the emailed group spends 6,000 euros over the next week and the holdout group spends 5,000 euros. The holdout stands in for what the emailed group would have spent without the email, so the estimated effect is 6,000 - 5,000 = 1,000 euros, or 1,000 / 1,000 = 1 euro per emailed customer. If the emailed group had instead been the 1,000 best customers, the same subtraction would mix the effect of the email with the difference between the groups.

Why Counterfactual matters for ecommerce

The causal impact of a campaign is the gap between what happened and what would have happened without it, so every incrementality number rests on a counterfactual that somebody had to supply. A report of observed conversions shows only what happened. The stand-in decides how far the number can be trusted: a 2019 Marketing Science paper compared observational methods with randomized advertising experiments and found that the observational methods often failed to produce the same effects, even when they controlled for extensive demographic and behavioral variables. Asking which counterfactual a figure used, and what it assumed, shows how much of the number is measured and how much is modeled.

How to use Counterfactual

1. Name the two worlds in one sentence: revenue in the campaign week with the campaign, and revenue in the same week without it. Fix the unit (customer, region or day), the outcome (orders or revenue in Shopify) and the window before the campaign starts.
2. Choose the stand-in for the missing world and write down what it assumes. A randomized holdout assumes the split was random. A comparison group assumes the two groups differ only in what you adjusted for. A forecast from earlier weeks assumes the pattern before the campaign would have continued.
3. Compute the effect as observed minus counterfactual. In a spreadsheet, take the average outcome per customer in the treated group, subtract the average in the holdout, and multiply by the number of treated customers.
4. Check the stand-in before trusting it. Compare the groups on prior orders and revenue by averaging each variable separately for treated and holdout customers, and compare a forecast with actuals in a period when no campaign ran.
5. Look for leakage. If holdout customers can still get the offer, or people in the two groups influence each other, the holdout is not a clean stand-in. The potential outcomes framework assumes each person's outcome depends on their own treatment assignment only.
6. Judge the result. Pass: the groups looked alike before the campaign, nothing leaked into the holdout, and the report names the stand-in it used. Fail: the groups differed before the campaign, or the holdout was exposed. Then the subtraction mixes the effect with those differences, and the figure should be reported as a comparison, not as the effect.

Formula

Effect for unit i = Y1_i - Y0_i

Where:
- Y1_i: the outcome unit i has if it is treated, such as a customer's spend with the campaign
- Y0_i: the outcome unit i has if it is not treated, such as the same customer's spend without it
- Effect for unit i: the gap between the two outcomes for the same unit

Only one of Y1_i and Y0_i is ever observed for a given unit, and the other is its counterfactual.

Common mistakes

1. Using the weeks before the campaign as the counterfactual. Season, trend and other promotions move revenue with no campaign at all, and a model built for this purpose has to handle trend, seasonal swings and variables whose influence changes over time, as a 2015 paper on marketing interventions does.
2. Letting the comparison group differ from the treated group before treatment. Cunningham calls the resulting gap selection bias: two groups with different distributions of a potential outcome. If the treated group was chosen for expected response, the comparison group is not its counterfactual.
3. Letting treatment spill into the holdout. If holdout customers see the same offer through another channel or through people in the treated group, their outcome no longer reflects no treatment. The potential outcomes framework assumes no interference between units.
4. Claiming that a single order was caused by an ad. The effect for one customer needs both of that customer's outcomes, and Cunningham writes that the unit-level effect cannot be calculated, so estimates describe groups of customers, not individual orders.
5. Reporting a modeled counterfactual without its assumptions. A forecast or a matched group is a model, and the 2015 paper itself discusses the strengths and limitations of its approach for settings where a randomized experiment is unavailable.

Frequently asked questions

• What is a counterfactual in simple terms?

  It is what would have happened if something had gone differently, such as a customer's purchases had they not been sent a campaign. It is never observed, because each customer either received the campaign or did not.
• Why can't a counterfactual be observed?

  Each unit receives one treatment, so only one of its two potential outcomes is ever seen. Cunningham's textbook calls this the fundamental problem of causal inference, cites Holland (1986) for it, and says a bigger dataset cannot fix it, since at least half of the outcomes needed are always missing.
• How is a counterfactual estimated for a marketing campaign?

  With a stand-in: a randomized holdout, a comparison group, or a model that forecasts the outcome without the campaign from earlier data. Randomization balances observed and unobserved variables in large samples, while the other two rest on assumptions that should be stated with the result.
• Is a control group the same as the counterfactual?

  No. The counterfactual is the treated group's own outcome without treatment, which is never observed. A control group is a stand-in for it, and it works only if that group would have behaved like the treated group without treatment, which random assignment is meant to secure.

Further reading

• Causal Inference: The Remix, Chapter 4: Potential Outcomes and Randomization · Cunningham
• Counterfactual Theories of Causation · Stanford Encyclopedia of Philosophy
• Inferring causal impact using Bayesian structural time-series models · arXiv (Annals of Applied Statistics)
• A Comparison of Approaches to Advertising Measurement: Evidence from Big Field Experiments at Facebook · Marketing Science (RePEc)
• The Effect, Chapter 10: Treatment Effects · Huntington-Klein

Related terms

• Causal InferenceCausal inference is the set of methods for estimating how much an outcome changes because of a cause, rather than…
• ColliderA Collider is a variable in a directed acyclic graph (DAG) that two or more other variables cause. Conditioning on a…
• ConfoundingConfounding is a distortion of the estimated treatment effect when a third variable, a confounder, associates with both…
• Interaction EffectAn Interaction Effect occurs when one variable's effect on an outcome depends on another variable's level.
• Potential Outcomes FrameworkThe potential outcomes framework defines a causal effect as the difference between the outcome a unit would have with a…
• Treatment EffectTreatment Effect is the causal impact of an intervention on an outcome. In marketing, this means the change in a metric…

Keep reading

Go deeper: Causal attribution, explained.

• Best Causal Inference Tools for Marketing Attribution
• Causal Inference for Marketing Teams: A Non-Technical Guide
• What Is Causal Inference and Why Every Marketer Needs It
• Causal Inference: The Future of Marketing Attribution
• Causal Inference Vs Rule Based Attribution
• MMM vs MTA vs Causal Inference

All glossary terms · More in Causal Inference

Have an audience of ecommerce operators? Earn 20% recurring commission by recommending the €99 read.