Counterfactual modeling

This document details how the Meridian GeoX framework utilizes counterfactual modeling to accurately measure the true incrementality of your marketing interventions.

Average treatment effect on treated

Counterfactual or potential outcome modeling is a standard approach in causal inference. It's essential to measure the average treatment effect on treated (ATT) or the true incremental impact of your marketing interventions.

Mathematically, the expected incremental impact on conversion outcome \( Y \) following a marketing intervention is defined as:

$$ATT = E[Y (1) - Y (0)|D = 1]$$

In the equation:

  • \( D = 1 \) denotes the treated geos based on study geo assignment rule.
  • \( Y (1) \) is the potential outcome of these geos with treatment (factual).
  • \( Y (0) \) is the potential outcome without treatment (counterfactual).

During the test period of GeoX, only \( Y (1) \) can be observed. \( Y (0) \) is unobserved and is predicted by GeoX's modeling approaches.

Machine learning problem under business-as-usual

The key assumption in GeoX is that without a newly added marketing intervention, all geos should all be under business-as-usual (BAU) even though their geo level conversion data may experience some natural fluctuations such as seasonality.

Using the BAU condition, GeoX builds a machine learning model that predicts the conversion of treated geos from the control pool. This model can further help predict the counterfactual metric \( Y (0) \) using the control geos which remain at BAU even after marketing intervention has been placed on treated geos. Thus, the loss function in the learning problem is calculated as:

$$\sum_{t \in \text{pretest}} \|f(Y_{t, T}(0)) - g(Y_{t, C}(0))\|_2{ }^2$$

In the equation:

  • \( Y_{t,T} \) and \( Y_{t,C} \) are vectors of conversion at time \( t \) of all treated and all control geos, respectively. The loss function sums up the L2 norm of the residuals by comparing the difference between treated units and control units with only pretest data.
  • Functional \( f \) is usually defined manually, taking an average conversion at the same time \( t \) across all treated geos.
  • Functional \( g \) can be learned from pretest data by minimizing the total loss function. Different analysis methodologies come up with different values of \( g \). In time-based regression (TBR), \( g \) is a linear transformation applied on the average conversion across control geos.

Like typical machine learning problems, sometimes it's quite meaningful to add the L1 or L2 regularization term in the loss function.

The core of the analysis module is that GeoX predicts what your conversion counts or revenue would've been for the treatment group if you hadn't made any marketing interventions, and then estimates the incremental effect accordingly. Examples of marketing interventions include new campaign activation, shut off media, and increase in spend.