Potential Outcomes Framework

What is Potential Outcomes Framework?

In the potential outcomes framework, each unit has one outcome it would show if treated and another it would show if not. The causal effect on that unit is the difference between the two. Only the outcome that matches the treatment actually received is observed, and the other exists only as a counterfactual, so no unit-level effect is ever measured. This is the fundamental problem of causal inference, and Scott Cunningham's online textbook Causal Inference: The Remix notes that more data cannot overcome it, because the dataset will always be too small by half. Group averages can be estimated, which is why the usual targets are averages such as the average treatment effect.

The framework goes back to Jerzy Neyman, whose analysis of agricultural field experiments used an array of unknown potential yields, one for each crop variety on each plot. Donald Rubin extended it to observational studies, and the Stanford Encyclopedia of Philosophy credits Neyman (1923) with pioneering it and Rubin (1974) and Robins (1986), among others, with developing it. It is also called the Rubin causal model or the Neyman-Rubin causal model. It is not the only formalism: an essay by Guido Imbens compares it with directed acyclic graphs and says much of the work in economics is closer in spirit to potential outcomes.

Estimates need an assignment mechanism and assumptions. When treatment is assigned independently of potential outcomes, as in a randomized test, the groups have the same potential outcomes on average, so differences in group means estimate average effects. Otherwise the framework relies on unconfoundedness given covariates, overlap between treated and untreated units, and SUTVA: one unit's treatment does not affect another unit's outcome, and the treatment is the same for every unit that gets it. Unconfoundedness is argued from domain knowledge, not confirmed by the data.

For illustration, suppose four customers have these orders in the next month with and without a retargeting ad, written as with / without: A 1 / 1, B 1 / 1, C 1 / 0 and D 0 / 0. Only C is moved by the ad, so the average effect over the four is (0 + 0 + 1 + 0) / 4 = 0.25 orders. Suppose the ad goes to A and B, who were already going to buy. Then the observed mean is 1 order for the treated and 0 for the untreated, a gap of 1 - 0 = 1, while the effect on the treated is (0 + 0) / 2 = 0. The gap measures who was chosen, not what the ad did.

Why Potential Outcomes Framework matters for ecommerce

Orders that follow exposure to a campaign are not the campaign's effect. The effect is the orders with the campaign minus the orders the same customers would have placed without it, and the second number is never observed. The framework makes that gap explicit and shows what has to stand in for the missing number: a randomized holdout, a comparison group argued to be comparable, or a model with stated assumptions. In the eBay paid-search experiment described in Cunningham's online textbook Causal Inference: The Remix, customers who clicked ads on searches for the company's own name were far more likely to buy than others, yet the randomized comparison found an effect close to zero, which the paper's authors explain by those users already intending to reach the site. A 2019 Marketing Science paper on Facebook ad experiments found that observational methods often failed to match randomized results.

How to use Potential Outcomes Framework

1. Write the question in potential-outcomes form. Name the unit (customer, session or region), the treatment (exactly what changes, such as receiving a discount email) and the outcome (orders or revenue in a fixed window). For each unit the two outcomes in play are the outcome with the treatment and the outcome without it.
2. Say which outcome you observe for each unit and which you must stand in for. Treated customers show the outcome with the treatment. The outcome without it has to come from a random holdout, a comparison group argued to be comparable, or a model.
3. Choose the stand-in and state its assumption. A random holdout needs only that assignment was random. A comparison group, such as customers who did not click, needs unconfoundedness: nothing that drives both exposure and purchases is left out. Cunningham marks assignment that sorts people by expected gain, such as ads targeted to people who will benefit, as a source of selection bias.
4. Check for interference and for hidden variation in the treatment. Ask whether one customer's treatment could change another's outcome, for example through a shared household, a shared code or a regional campaign, and whether the treatment differs between treated customers, for example by send time or offer. If interference is possible, randomize at a higher level such as household or region, as the school example in the guide for grf, an R package for forest-based causal inference, does by randomizing whole schools. If the treatment varies, define it more narrowly.
5. Pass: you can say in one sentence what stands in for the missing outcome and why it is credible. Fail: the only evidence is orders after exposure, or the exposed were chosen by intent or by an algorithm. Then describe the number as an association, not an effect.

Formula

Effect for unit i = Y_i(1) - Y_i(0)
Observed outcome: Y_i = D_i x Y_i(1) + (1 - D_i) x Y_i(0)

Where:
- Y_i(1): the outcome of unit i if the unit is treated
- Y_i(0): the outcome of unit i if the unit is not treated
- D_i: equals 1 if the unit received the treatment and 0 if not
- Y_i: the outcome actually observed for unit i

Only one of the two potential outcomes is observed for each unit, so the effect for a single unit cannot be computed from data.

Common mistakes

1. Counting orders after exposure as the effect. Those orders show the outcome with the treatment for exposed people and say nothing about the outcome without it. In the eBay case the estimated return was virtually all selection bias. The cost is crediting a campaign for purchases that would have happened anyway.
2. Comparing exposed with unexposed customers and calling the gap the effect. If treatment goes to people expected to benefit, or to people who already intend to buy, the gap includes selection, and Cunningham notes the sign of that bias can go either way. The cost is a number that does not measure the campaign.
3. Expecting more data to supply the missing outcome. The missing half does not shrink as the dataset grows, and only a design such as randomization, or an assumption, can replace it. The cost is waiting for a larger dataset to settle a question that only the design can settle.
4. Ignoring interference. If a treated customer's code or recommendation reaches a control customer, the control no longer shows the outcome without the treatment. The cost is an estimate that does not match the effect of treating versus not treating.
5. Treating the framework as an estimator. It defines the effect and lists the assumptions, and it estimates nothing by itself. Without randomization an estimate is only as credible as its assumption. The cost is false confidence in a number.

Frequently asked questions

• What is the potential outcomes framework in simple terms?

  It treats every customer as having two possible outcomes, one if they get a treatment such as an ad and one if they do not. The causal effect is the difference between them. Only one outcome is ever seen for each customer, so effects are estimated by comparing groups.
• What is the fundamental problem of causal inference?

  For any one unit, only the outcome under the treatment actually received is observed, and the other outcome never occurs, so the unit-level effect cannot be measured directly. Cunningham's online textbook Causal Inference: The Remix adds that more data does not fix this, because the dataset is always too small by half. Averages across groups can still be estimated.
• Is the potential outcomes framework the same as the Rubin causal model?

  Mostly. The Rubin causal model, also called the Neyman-Rubin causal model, is the approach built on potential outcomes. Wikipedia says Neyman first proposed the framework in 1923 and Rubin extended it to observational studies. Graphical models are a separate formalism, which Imbens compares with it.
• How does the potential outcomes framework apply to marketing measurement?

  Treat exposure to an email or ad as the treatment and orders as the outcome. The effect is the orders with exposure minus the orders without it, and the second is never observed, so a randomized holdout or an argued comparison group has to stand in for it. Counting orders after exposure does not do that.

Further reading

• Causal Inference: The Remix, chapter 4: Potential Outcomes and Randomization · Scott Cunningham
• Causal Models · Stanford Encyclopedia of Philosophy
• Potential Outcome and Directed Acyclic Graph Approaches to Causality: Relevance for Empirical Practice in Economics · Guido W. Imbens, arXiv
• On the Application of Probability Theory to Agricultural Experiments. Essay on Principles. Section 9 · Splawa-Neyman, Dabrowska and Speed, Statistical Science

Related terms

• Causal ForestsCausal forests are a random-forest method that estimates how a treatment's effect differs between units with different…
• Causal InferenceCausal inference is the set of methods for estimating how much an outcome changes because of a cause, rather than…
• Causal PathwayA Causal Pathway is the sequence of events through which a cause produces an effect.
• CounterfactualA counterfactual is what would have happened under conditions that did not occur, such as a customer's outcome had they…
• InterventionAn Intervention is an action taken to produce a change in an outcome.
• Treatment EffectTreatment Effect is the causal impact of an intervention on an outcome. In marketing, this means the change in a metric…

Keep reading

Go deeper: Causal attribution, explained.

• Best Causal Inference Tools for Marketing Attribution
• Causal Inference for Marketing Teams: A Non-Technical Guide
• What Is Causal Inference and Why Every Marketer Needs It
• Causal Inference: The Future of Marketing Attribution
• Causal Inference Vs Rule Based Attribution
• MMM vs MTA vs Causal Inference

All glossary terms · More in Causal Inference

Have an audience of ecommerce operators? Earn 20% recurring commission by recommending the €99 read.