Causal Forests and Average Treatment Effects in Marketing

A causal forest estimates the conditional average treatment effect (CATE): how much a treatment, such as seeing an ad, changes an outcome for customers who share the same characteristics. The average treatment effect (ATE) is the mean of that change over the group you care about. On purchase data you did not randomise, both numbers rest on the assumption that exposure is unconfounded given the covariates you recorded, and in 663 Facebook experiments two non-experimental methods did not recover the randomised result: the median upper-funnel lift was 29% in the trials and 83% with double/debiased machine learning.

What does a causal forest estimate?

Every customer has two potential outcomes: what they would spend with the treatment and what they would spend without it. Only one of the two is observed for any customer, so the effect on one customer is never seen and has to be estimated from groups. A causal forest estimates that effect for each customer profile. For a binary, unconfounded treatment, the grf documentation defines the target as the expected gap between the two outcomes for customers with covariates x:

Y(1) = outcome with the treatment, Y(0) = outcome without it
tau(x) = E[Y(1) - Y(0) | X = x]

Wager and Athey built the method by extending Breiman's random forest and show that, under unconfoundedness, its estimates are consistent for the true treatment effect. By default grf uses honest splitting: one half of each subsample chooses the splits and the other half fills the leaves. The method is non-parametric: it assumes no formula for how the effect varies across customers, which is why it suits heterogeneous treatment effects.

What is the average treatment effect, and how does grf report it?

The average treatment effect is the mean of the gap between the two outcomes over a chosen group, and the choice of group changes the question. The grf documentation lets you average over all units, over the treated, or over the controls:

all:      E[Y(1) - Y(0)]
treated:  E[Y(1) - Y(0) | W = 1]
control:  E[Y(1) - Y(0) | W = 0]

The first is what the treatment would do for everyone, the second what it did for those who received it. The grf package does not simply average the forest's predictions. Its algorithm reference says that plugging them into an estimator built for the ATE is more accurate than averaging the personalised effects. average_treatment_effect() offers augmented inverse-propensity weighting (AIPW) or targeted maximum likelihood estimation (TMLE) and returns the estimate along with a standard error.

When can an ecommerce brand read the estimate as causal?

The grf documentation names two conditions, both about how customers came to be treated. The first is unconfoundedness: given the covariates you recorded, who was treated must be unrelated to what they would have bought. The forest's guarantees are stated under it, so it is an assumption about your business, not something the forest delivers. The second is overlap: grf's reference says propensity scores must be bounded away from zero and one for the conditional effects to be identified. Every kind of customer needs some chance of being treated and some of not: where a segment has no untreated customers, there is nothing to compare. For poor overlap, the grf documentation points to an overlap-weighted target, which does not divide by estimated propensities.

What does the evidence say about observational data?

Gordon, Zettelmeyer, Bhargava and Chapsky used 15 U.S. advertising experiments at Facebook, with 500 million user-experiment observations, and report that the observational methods often failed to produce the same effects as the randomised experiments, even after conditioning on extensive demographic and behavioral variables (Marketing Science, 2019).

Gordon, Moakler and Zettelmeyer analysed 663 experiments with over 5,000 user-level features, more than most advertisers can access. The median lifts in the randomised trials were 29%, 18% and 5% for upper, middle and lower funnel outcomes. Double/debiased machine learning gave median lifts of 83%, 58% and 24%, and stratified propensity score matching gave 173%, 176% and 64%. The authors report that they were unable to reliably estimate an ad campaign's causal effect.

A holdout has limits too. Lewis and Rao report on twenty-five large field experiments with major U.S. retailers and brokerages: the median confidence interval on return on investment was over 100 percentage points wide, and informative experiments can easily require more than 10 million person-weeks (Quarterly Journal of Economics, 2015).

How strong is this? All three papers are peer reviewed, and the 2023 paper tests two other estimators. No abstract read for this post reports a causal forest on a store's data, so the evidence shows how observational estimates can fail, not how often a forest does.

How can I check my own data in a spreadsheet?

Take a past campaign you want to judge, such as an email send, a discount or an ad audience.

1. In Shopify, open Orders, click Export and choose Orders by date, covering the campaign and an equal window before it. Shopify's CSV puts extra line items of an order on separate lines, so count distinct order numbers in the Name column, not rows.
2. Make one row per customer email with three columns: exposed (yes or no, from your send list or audience), orders before the campaign, orders during it.
3. Overlap check: group customers by prior orders (none, one, two or more) and count exposed and unexposed customers in each group. Pass: every group has both. Fail: a group has nobody exposed or nobody unexposed, so no estimate for that group can be trusted.
4. Balance check: compare average prior orders for exposed and unexposed customers. Pass: the gap is small next to the gap you see after the campaign. Fail: exposed customers already ordered more before it, so part of the later gap is who was chosen, not what the campaign did.
5. For the next send, randomise: put =RAND() beside each customer, sort, hold out the lowest-ranked slice, send to the rest and compare orders per customer after the window. Pass: the two groups look alike on prior orders (the balance check) and nobody in the holdout got the message another way. Fail: prior orders differ, so the split was not random, or holdout customers saw the campaign elsewhere, which shrinks the measured gap.

With few customers in each group the gap can be too noisy to read, the problem Lewis and Rao describe.

Later, once a holdout gives you a benchmark, a causal attribution tool like Causality Engine can read a GA4 export in 1 to 2 minutes and show what each GA4 channel group caused next to last-click. It works on channel groups, not campaigns, and a GA4 export is observational data, so check its read against the holdout result.

Sources, 6 October 2026: Estimation and Inference of Heterogeneous Treatment Effects using Random Forests (Journal of the American Statistical Association, 2018); Recursive Partitioning for Heterogeneous Causal Effects (arXiv, 2015); Causal forest (grf documentation); average_treatment_effect (grf documentation); The GRF Algorithm (grf documentation); A Comparison of Approaches to Advertising Measurement: Evidence from Big Field Experiments at Facebook (Marketing Science, 2019); Close Enough? A Large-Scale Exploration of Non-Experimental Approaches to Advertising Measurement (Marketing Science, 2023); The Unfavorable Economics of Measuring the Returns to Advertising (Quarterly Journal of Economics, 2015); Exporting orders (Shopify Help Center).

Related answers

• Average treatment effect
• Causal forests
• Heterogeneous treatment effects
• Incrementality testing
• Causal attribution