Skip to content

Matching

Matching estimates a treatment effect from observational data by comparing treated units with untreated units that look alike on chosen variables. It can adjust only for variables that were measured, and it assumes they close every back door.

Three platforms, three answers, one budget to defend. Your GA4 export already holds the answer, and the first finding is free and never leaves your browser. Get your first finding free

By , Founder & CEOUpdated 7 min read

What is Matching?

In observational data, units that received a treatment can differ from those that did not in ways that also move the outcome. Matching responds by building comparison groups whose matching variables look alike, which The Effect presents as a way of closing back doors. Stuart's 2010 review defines it broadly as any method that aims to balance the covariate distributions of the treated and control groups, so that the comparison comes as close as it can to a randomized experiment. It is a design step: matches are chosen without the outcome, and outcomes are compared afterwards.

Methods differ in how they define similar. Distance matching treats observations as similar when their matching variables have similar values, for example by Mahalanobis distance. Propensity score matching treats them as similar when their chances of being treated were the same; the propensity score is a unit's probability of being treated, given its observed covariates. Exact matching requires identical values, coarsened exact matching first sorts continuous variables into bins, and a caliper sets the largest gap a match may have before it is rejected.

The common targets are the average effect on the treated and the average treatment effect; the example below estimates the first. Two assumptions carry the method. Unconfoundedness, also called ignorability or selection on observables, says that given the matching variables, treatment is independent of the potential outcomes. Common support says that treated and untreated units exist at the same values of those variables, so treated units with no similar untreated unit have to be dropped, which changes whom the estimate describes. Anything unmeasured stays unbalanced, and a propensity score does not change that.

Parts of the method are contested. King and Nielsen argue that propensity score matching often leaves groups more imbalanced and raises bias, and they suggest other matching methods. In advertising, a 2023 Marketing Science paper analyzed 663 large-scale experiments at Facebook and reported that neither stratified propensity score matching nor double machine learning performed well, with user-level data richer than most advertisers can get.

For illustration, suppose three customers received a discount email and three did not, and each emailed customer is matched to the non-emailed customer with the closest prior-year spend. Suppose the emailed customers spent 52, 39 and 26 euros next month and their matches spent 40, 30 and 20 euros. The matched differences are 12, 9 and 6, so the estimated effect on the emailed customers is (12 + 9 + 6) / 3 = 9 euros. If the emails went to customers whose interest in the brand was rising, which no matching variable captures, the matches do not remove that difference and the 9 euros still contains it.

Why Matching matters for ecommerce

Outside an experiment, customers who saw a campaign can differ from those who did not in ways that also drive purchases, so a raw comparison mixes the campaign's effect with those differences. Matching builds a fairer comparison from variables such as order history, recency and acquisition channel. Its limit matters as much as its use: it balances only what was matched on. A 2019 Marketing Science paper reported that observational methods often failed to produce the same effects as randomized advertising experiments, even when they controlled for extensive demographic and behavioral variables. A matched estimate is a comparison with a named assumption, and it deserves a check against a holdout when one exists.

How to use Matching

  1. State the comparison in one sentence: for example, customers who received a discount email versus customers who did not, with revenue in the weeks after the send as the outcome.
  2. List variables measured before the send that plausibly affect both who was emailed and the outcome, such as order count, revenue to date, days since the last order, acquisition channel and country. Leave out anything measured after the send, because the email may have changed it.
  3. Pick how to define similar: exact matches on a few categories, a distance such as Mahalanobis on several variables, or the propensity score. Set a caliper so poor matches are dropped, and decide how many matches each emailed customer gets. Make these choices without looking at the outcome.
  4. Check overlap. Emailed customers with no similar non-emailed customer are dropped, so count them and note who they are. The estimate then describes the customers who matched well, not everyone who was emailed.
  5. Check balance. Compare the means of every matching variable in the two matched groups using standardized differences, not significance tests. If a variable is still far apart, change the distance or the caliper and match again.
  6. Compare outcomes between the matched groups, which estimates the effect on the emailed customers. Pass: balance is good, few emailed customers were dropped, and the estimate holds when you change the matching variables or the method. Fail: balance is poor, many were dropped, or the estimate swings. Then report the figure as a comparison, not as the effect, and run a randomized holdout.

Formula

e(x) = P(T = 1 | X = x)

ATT estimate = (1 / n) Σ_i (Y_i - Y_matched(i))

Where:
- e(x): the propensity score, the probability of treatment for a unit with covariate values x
- T: whether a unit is treated, yes or no
- X: the covariates measured before treatment
- n: the number of treated units kept after matching
- Y_i: the outcome of treated unit i
- Y_matched(i): the average outcome of the untreated units matched to treated unit i

The first line is the score that propensity score matching uses to judge similarity, and the second averages, over the treated units, the gap between each unit's outcome and the outcome of its matches.

Common mistakes

  1. Matching on variables measured after treatment. Stuart says variables the treatment may itself have changed should stay out of the matching. If the email changed a customer's site visits, matching on visits after the send balances away part of what the email did.
  2. Assuming matching handles what was not measured. It balances only the matching variables, and Stuart notes that any non-experimental study can be criticized for unobserved variables tied to both treatment and outcome. In the 2023 advertising study, neither stratified propensity score matching nor double machine learning performed well against experiments.
  3. Skipping the balance check, or judging it with p-values. Stuart advises against t-tests and other significance tests as balance measures, because a test result moves with sample size, not only with balance. A matched sample can look similar overall and still be far apart on a variable that matters.
  4. Dropping unmatched emailed customers without saying so. Stuart notes that discarding treated units changes the group the results apply to, so the headline number can describe a narrower set of customers than the campaign reached.
  5. Using propensity score matching by default. King and Nielsen show it can increase imbalance and bias, and The Effect advises against using propensity scores to accept or reject matches, preferring weights. Compare it with exact, coarsened exact or distance matching on the same data.

Frequently asked questions

  • What is matching in causal inference?
    Matching builds a comparison group from untreated units that resemble the treated units on chosen variables, so the groups are balanced before outcomes are compared. It is a design step that does not use the outcome, and it can adjust only for the variables you matched on.
  • What is propensity score matching, and is it the best way to match?
    The propensity score is a unit's probability of being treated, given its observed covariates, and propensity score matching pairs units with similar scores. It is widely used and contested: King and Nielsen show in a 2019 paper that it can leave groups more imbalanced and raise inefficiency, model dependence and bias, and they suggest other matching methods.
  • Can matching replace a randomized experiment?
    No. Matching balances only the variables used. A 2023 Marketing Science paper found that stratified propensity score matching and double machine learning did not perform well against Facebook advertising experiments, even with user-level data richer than most advertisers can obtain. A randomized holdout needs no matching.
  • What is common support in matching?
    Common support means that treated and untreated units exist with the same values of the matching variables or propensity scores. Treated units with no comparable untreated unit are dropped, so the estimate then describes only the units that matched well.

Further reading

Your platforms guess.
We run the math.

Upload a GA4 export and see what each channel caused, next to last-click, in 1–2 minutes. The read is yours to keep.

Get your first finding free

The free finding runs in your browser. Your file never leaves it. The full read is €99, refundable within 30 days, no questions asked. Prices exclude VAT.