Skip to content

Causal Inference

Causal inference is the set of methods for estimating how much an outcome changes because of a cause, rather than alongside it. It works by comparing what happened with an estimate of what would have happened without the cause.

Three platforms, three answers, one budget to defend. Your GA4 export already holds the answer, and the first finding is free and never leaves your browser. Get your first finding free

By , Founder & CEOUpdated 7 min read

What is Causal Inference?

A causal claim says that, had the cause been absent, the outcome would have differed by some amount. Causal inference supports such claims with data. It differs from describing association because it asks how an outcome responds when a cause is changed, not only whether the two move together. Wikipedia notes that this is difficult and that scientists debate the proper way to do it.

The core difficulty is that half of the comparison is never observed. For one customer, day or market, only the outcome under the treatment actually received appears; the other outcome, the counterfactual, is missing. Cunningham's textbook calls this the fundamental problem of causal inference and describes it as a missing-data problem, so the effect on a single unit cannot be calculated. Effects are estimated as averages over groups, and Huntington-Klein's textbook notes that each person has their own treatment effect, which the data do not show.

An average is credible when the groups compared would have had the same outcomes without the treatment. Random assignment produces such groups on average, because chance, not anything tied to the outcome, decides who is treated. Without it, the comparison rests on assumptions the data cannot confirm: Cunningham writes that unconfoundedness is not testable, and that parallel trends cannot be tested because it involves a counterfactual that cannot be observed. Textbooks on observational causal inference cover designs such as matching, instrumental variables, regression discontinuity and difference-in-differences.

Several frameworks are in common use. The potential outcomes framework, developed by Rubin and coauthors building on work by Neyman, defines an effect as a difference between potential outcomes. The graphical approach uses directed acyclic graphs and is associated with Pearl; Wikipedia lists further frameworks. Imbens compares the first two, and the Stanford Encyclopedia of Philosophy traces causal modeling to the statistical work of the 1920s, especially Sewall Wright's.

For illustration, suppose a store emails 9,000 of its 10,000 customers and holds back a random 1,000. Suppose 450 of the emailed customers buy that month, 450 / 9,000 = 5%, and 30 of the held-back customers buy, 30 / 1,000 = 3%. Then the estimated effect of the email is 5% - 3% = 2 percentage points, or 9,000 x 0.02 = 180 orders, where a report crediting the email with every order that follows it would count 450.

Why Causal Inference matters for ecommerce

A budget decision is a causal claim: spend more here and revenue rises by this much. Shopify documents its attribution models as rules for allocating credit among the channels a customer touched, which says who gets credit for an order, not how many orders the spending added. Two sets of experiments show how far the two can differ. A 2019 Marketing Science paper compared Facebook ad experiments with observational methods and found that the observational estimates often failed to reproduce the experimental ones, even with extensive controls. Field experiments at eBay, published in Econometrica in 2015, reported that returns from paid search were a fraction of non-experimental estimates, and that brand keyword ads had no measurable short-term benefit. The Facebook paper adds that unobservable factors make ad exposure endogenous, so who sees an ad is tied to things the data do not record, and that advertising's effect tends to be small, which makes it hard to measure.

How to use Causal Inference

  1. Write the question as a comparison. Name the action (an email send, a campaign, a price change), the outcome (Shopify orders or net sales), the unit (customer, day or region) and the period, then state what the outcome would have been without the action. That unseen number is what has to be estimated.
  2. List what else changed in the same window and could move the outcome: promotions, price changes, stock-outs, other channels, the season. Put the dates in a spreadsheet next to the outcome. Any item that moves together with the action is a confounder.
  3. Choose the strongest comparison available. The best is a random holdout: keep a random share of customers out of an email, or a random set of regions out of a campaign, and compare the two groups. If randomizing is not possible, build the comparison from the data, for example against a comparison group or the earlier trend, and write down the assumption it needs.
  4. Compute the effect as the average outcome of the treated group minus the average outcome of the comparison group, and report it with a confidence interval, not as a single number.
  5. Check the comparison before trusting it. With a random split, the groups should look alike on past orders. Without randomization, check that the groups moved together before the action, and treat that as evidence for the assumption, not as a test of it.
  6. Read the result. Pass: the groups matched before the action, the estimate is large relative to its interval, and nothing else in the window lines up with the action. Fail: the groups differed beforehand, the interval spans both a useful effect and zero, or a promotion overlaps. A wide interval means the data cannot say, which is different from no effect.

Formula

tau = Y1 - Y0

ATE = E[Y1 - Y0]

Where:
- Y1: the outcome a unit would have with the treatment
- Y0: the outcome the same unit would have without it
- tau: the effect on one unit, never observed, because only one of the two outcomes occurs
- ATE: the average treatment effect, the mean of tau over a population
- E[ ]: the average over that population

An experiment estimates ATE by comparing group averages, which works when the groups would have had the same outcomes without the treatment.

Common mistakes

  1. Crediting an action with every order that follows it. Credit rules and before-and-after counts include orders that would have happened anyway. At eBay, comparing buyers who saw branded search ads with those who did not suggested a large return, and experiments later found no measurable short-term benefit from brand keyword ads.
  2. Adding controls and treating the problem as solved. Controls handle only what is measured. In the Facebook comparison, observational methods often failed to match the experiments even after conditioning on extensive demographic and behavioral variables.
  3. Reading a before-and-after change as the effect. Anything else that changed in the window, such as a sale or the season, is credited to the action. The comparison needs a control group or a model of the earlier trend.
  4. Treating a wide interval or a non-significant result as evidence of no effect. A precisely estimated zero is a finding; an imprecise estimate is not. The Facebook paper notes that advertising effects tend to be small, which makes them hard to measure.
  5. Applying an average effect to every customer. The average treatment effect is the mean of effects that differ across units, and the data show group averages only, so a campaign can help some customers and not others.

Frequently asked questions

  • What is the difference between causal inference and correlation?
    Correlation describes how two things move together; causal inference asks how the outcome responds when the cause is changed. That needs a comparison with what would have happened otherwise, because two series can move together for reasons unrelated to each other, such as the season.
  • What is the fundamental problem of causal inference?
    For any one customer, day or market, only the outcome under the treatment it actually received is observed. The other outcome, the counterfactual, is missing, so the effect on that unit cannot be calculated. Effects are estimated as averages over comparable groups instead.
  • Can you do causal inference without an experiment?
    Sometimes, using designs such as matching, instrumental variables or difference-in-differences, each resting on assumptions the data cannot confirm. Results can differ from experiments: a 2019 paper on Facebook ad experiments found observational methods often failed to reproduce them.
  • Is attribution the same as causal inference?
    No. Shopify documents its attribution models, such as last click and first click, as rules for allocating credit among the channels a customer touched. Causal inference estimates what the marketing added by comparing with what would have happened without it.

Further reading

Your platforms guess.
We run the math.

Upload a GA4 export and see what each channel caused, next to last-click, in 1–2 minutes. The read is yours to keep.

Get your first finding free

The free finding runs in your browser. Your file never leaves it. The full read is €99, refundable within 30 days, no questions asked. Prices exclude VAT.