Counterfactual modeling

This document details how the Meridian GeoX framework utilizes counterfactual modeling to accurately measure the true incrementality of your marketing interventions.

Average treatment effect on treated

Counterfactual or potential outcome modeling is a standard approach in causal inference. It's essential to measure the average treatment effect on treated (ATT) or the true incremental impact of your marketing interventions.

Mathematically, the expected incremental impact on conversion outcome \( Y \) following a marketing intervention is defined as:

$$ATT = E[Y (1) - Y (0)|D = 1]$$

In the equation:

  • \( D = 1 \) denotes the treated geos based on study geo assignment rule.
  • \( Y (1) \) is the potential outcome of these geos with treatment (factual).
  • \( Y (0) \) is the potential outcome without treatment (counterfactual).

During the test period of GeoX, only \( Y (1) \) can be observed. \( Y (0) \) is unobserved and is predicted by GeoX's modeling approaches.

Machine learning problem under business-as-usual

The key assumption in GeoX is that without a newly added marketing intervention, all geos should all be under business-as-usual (BAU) even though their geo level conversion data may experience some natural fluctuations such as seasonality.

Using the BAU condition, GeoX builds a machine learning model that predicts the conversion of treated geos from the control pool. This model can further help predict the counterfactual metric \( Y (0) \) using the control geos which remain at BAU even after marketing intervention has been placed on treated geos. Thus, the loss function in the learning problem is calculated as:

$$\sum_{t \in \text{pretest}} \|f(Y_{t, T}(0)) - g(Y_{t, C}(0))\|_2{ }^2$$

In the equation:

  • \( Y_{t,T} \) and \( Y_{t,C} \) are vectors of conversion at time \( t \) of all treated and all control geos, respectively. The loss function sums up the L2 norm of the residuals by comparing the difference between treated units and control units with only pretest data.
  • Functional \( f \) is usually defined manually, taking an average conversion at the same time \( t \) across all treated geos.
  • Functional \( g \) can be learned from pretest data by minimizing the total loss function. Different analysis methodologies come up with different values of \( g \). In time-based regression (TBR), \( g \) is a linear transformation applied on the average conversion across control geos.

Like typical machine learning problems, sometimes it's quite meaningful to add the L1 or L2 regularization term in the loss function.

The core of the analysis module is that GeoX predicts what your conversion counts or revenue would've been for the treatment group if you hadn't made any marketing interventions, and then estimates the incremental effect accordingly. Examples of marketing interventions include new campaign activation, shut off media, and increase in spend.

Assumptions and implications of violations

Most geo-experimentation methodologies used to establish counterfactual conversions—including Meridian GeoX—are formally grounded in sharing a common latent factor structure (Bai, 2009; Abadie, Diamond, and Hainmueller, 2010).

In plain terms, this framework assumes that the daily conversion volume in a given geo is shaped by broader, time-varying market forces, such as national media campaigns, seasonality, or macroeconomic trends (referred to as the common trend assumption). While these forces affect the entire market, individual markets can react with different levels of sensitivity. For example, the winter season might cause a dramatic drop in retail in New York while having almost no impact in Florida.

Under this premise, Meridian GeoX designs experiments by identifying treatment and control groups of geos whose combined sensitivities align and that have high predictive power for each other, thereby maximizing test precision, specifically by lowering the minimum detectable effect (MDE) at the design stage.

If the real data violates this assumption of a shared, stable latent factor structure among geos in the study, the analysis results may become underpowered (wide confidence intervals and an inconclusive result), miscalibrated (confidence intervals that are too narrow, so the false positive rate significantly exceeds your target significance level, alpha), or biased (the counterfactual systematically over- or under-predicts).

  • Shared: Treatment and control geos should be driven by the same underlying market forces so that the control group reliably carries information about the treatment group's counterfactual behavior. If geos don't share these common forces, candidate designs fit poorly, failing to reach the required pretest out-of-sample R-squared threshold (min_r2 in DesignConfig, default 0.8) or failing the pretest A/A test (detecting false incremental lift on historical data when no marketing intervention occurred). GeoX uses these pretest checks to screen out many poorly fitting designs during setup.
  • Stable: What is estimated in the pretest period should continue to hold during the test period, because the counterfactual model is fit on pretest data and extrapolated into the test window.

Common violations

Common violations include:

  • Spillover Effects (Media Bleed): Occurs when marketing targeted at a treatment geo spills into control geos, contaminating the baseline and underestimating incremental impact.
  • Local Shocks: Sudden, localized events (such as regional weather or local promotions) that affect only a few geos, breaking the shared trend. The timing determines the consequence:
    • During the pretest period: The shock degrades model fit and surfaces through the design-stage checks as an R-squared check failure or A/A test failure. These errors indicate the pretest data doesn't support a reliable counterfactual; the remedy is a cleaner pretest window.
    • During the test period: A disruption during the active test window alters market trends and distorts counterfactual predictions. Avoid scheduling experiments during windows with known regional disruptions, such as local promotions or major market changes.
  • Lack of Support (for example, Mega Geos): Occurs when a treated geo is so unique or large in scale (such as New York City or Tokyo) that its behavior can't be replicated by any combination of control geos.
  • Geo-Specific Trends (Random Walks): Some geos drift along their own path over time, on top of the market forces shared by all geos. Examples include a gradual local expansion or a slow local competitor entry. Statistically, each such geo follows its own random walk. These geo-specific movements build up over time instead of averaging out, so the relationship between treatment and control geos doesn't stay stable, violating the common trend assumption. Even if a candidate design passes pretest R-squared and A/A checks by chance, the accumulating drift during the test period causes counterfactual predictions to diverge, leading to overly narrow confidence intervals and an inflated false positive rate.

In practice, while violations of these assumptions aren't ideal, mild departures don't necessarily invalidate the results. For example, a small amount of spillover from commuters traveling across geo boundaries is often unavoidable. When the shared market signal still dominates and the design passes pretest R-squared and A/A checks, the impact is typically minor and the estimates remain practically useful.

By contrast, in the extreme case where geos share no common market forces and each geo moves along its own independent path (for example, each geo follows its own random walk, as in the right panel of the following figure), the common trend assumption is entirely violated and counterfactual modeling has no valid basis.

Two line charts of simulated geo-level conversions. Left: geos follow a common trend with different sensitivities. Right: each geo follows its own independent random walk.

Left: Geos share a common trend and react to it with different sensitivities. The common trend assumption holds. Right: Each geo follows its own independent random walk. The common trend assumption is violated.

Placebo screening and R-squared thresholds

Meridian GeoX quantifies uncertainty by running the analysis on placebo designs (see Robust inference), simulating pseudo-experiments on control geos to measure baseline market noise. To protect confidence interval accuracy, GeoX screens placebo design quality using an out-of-sample R-squared check (min_placebo_r2 in AnalysisConfig, default 0.6). When control markets are too volatile or diverse, GeoX raises a ValueError for insufficient valid placebo candidates rather than building intervals on poor matches.