Incremental Outcome, ROI, mROI & Response Curves

This section covers the key metrics of Meridian - Incremental Outcome, Return on Investment (ROI), marginal ROI (mROI) and response curves.

Quick Takeaways

Incremental Outcome, return on Investment (ROI), marginal ROI (mROI), and response curves are the tools that turn your model's findings into actionable business strategy. They help you answer the most critical marketing questions: "How well did my channels perform?" and "Where should I spend my next dollar?"

By understanding these metrics, you can identify your most efficient channels, understand current saturation levels, and optimize your budget to maximize your business outcomes. Response curves, in particular, provide a powerful visualization of how the incremental outcome responds to more spending, which is the foundation of data-driven budget allocation.

Marketing Example

Imagine you run an online shoe store. You spend \$10,000 on a video media channel. After running your Meridian model, you find that the channel caused \$25,000 in incremental sales.

• Incremental Outcome is the value your marketing caused. For example, your total sales were \$150,000, but Meridian estimates that without the campaign, sales would have been \$125,000. The Incremental Outcome is the difference: \$25,000.
• Your ROI is \$2.50 (\$25,000 sales / \$10,000 cost), meaning you earned \$2.50 for every dollar spent. (See Considerations for interpreting ROI and response curves for more details on how this is calculated).
• The response curve shows you how your sales would change at different spend levels. It shows that spending even more money yields progressively smaller returns.
• Your mROI is the return you'd get from some small increase in spend (for example, the next dollar). If your channel is nearing saturation, the mROI might be only \$0.80, signaling it's time to invest elsewhere.

Rule-of-Thumb Recommendation

• Use ROI to evaluate historical performance: It gives you a clear, overall grade for how effective your past spending was on a given channel.
• Use response curves to optimize future budgets: They visualize the point of diminishing returns, helping you understand how much you can invest in a channel before it becomes inefficient.
• Use mROI to evaluate saturation level: If the mROI is much lower compared to the ROI, then the channel is beginning to saturate at historical spend level. Channels with the highest mROI are the best for investing additional funds.

Comparison Table

Code Examples

Refer to Example - ROI, mROI & Response Curves in Meridian for working code examples.

Detailed Explanation

This section provides a deeper dive into the definitions and methodologies behind ROI, mROI, and response curves.

Incremental Outcome Explained

The foundation for ROI, mROI, and response curves is incremental outcome. This is the portion of your outcome, such as sales or conversions, that was caused by a specific marketing activity. Meridian calculates this by comparing the actual outcome to a counterfactual scenario where the marketing activity never happened. For paid media, the incremental outcome can be further contextualized by its spend in the following ways:

• The response curve estimates the incremental outcome at any given spend level.
• ROI is the incremental outcome at your historical spend level divided by the spend.
• mROI is the incremental outcome on your next dollar spent above the historical budget level.

How Response Curves Are Generated

A response curve visualizes the relationship between spend and incremental outcome for a single channel, assuming all other channels' spending remains the same.

Meridian generates this curve at different spending levels for a channel. It scales the channel's historical spend up or down by a factor (for example, from 1.2x the historical spend) and estimates the incremental outcome at each level. The historical distribution of spending over time and geography (the flighting pattern) is preserved during this scaling. This process reveals the point at which a channel becomes saturated and further investment yields diminishing returns.

Considerations for interpreting ROI and response curves

• Lagged effects: The ROI definition uses a channel's total cost over a specific period as the denominator. The numerator is the incremental outcome accrued during that same period. This numerator includes lagged effects from ads that ran before the period but excludes future effects from ads that ran during the period. Over a long time window (for example, one year), this has a minor effect on the ROI estimate. However, for shorter periods, the effect can be more meaningful.
• Extrapolation risk: Calculating the incremental outcome requires the model to estimate what would have happened if spend was zero. If you have always spent consistently on a channel, the model has little data for this zero-spend scenario and must extrapolate based on its learned assumptions. Extrapolation risk also affects incremental outcome estimation for points on the response curve that are greater than historical spend, and the risk increases the further out you go.

Mathematical Appendix

This section contains the mathematical definitions behind incremental outcome ROI, mROI and response curves.

Incremental outcome

We start with incremental outcome, as it serves as the underpinning of ROI, mROI, and response curves.

Causal inference framework behind incremental outcome

Meridian measures the causal effect of treatment variables on outcome. Outcome is typically revenue, but when the KPI is not revenue and revenue_per_kpi data is not available, then Meridian defines the outcome to be the KPI itself.

We define incremental outcome using the framework of causal inference. In causal inference, it is common to see notation like \(Y^{(1)}\) and \(Y^{(0)}\) representing potential outcomes under treatment and control counterfactual scenarios. Meridian is similar, but slightly more complex, because the treatment and control counterfactual scenarios are three-dimensional arrays across geos $g$, time $t$, and treatment variables $i$. We denote the treatment and control counterfactual scenarios as \(\left\{x^{(1)}_{g,t,i}\right\}\) and \(\left\{x^{(0)}_{g,t,i}\right\}\) respectively. Additionally, the potential outcomes are a two-dimensional array across geos and time. We denote the potential outcomes under the treatment and control counterfactual scenarios as \(\overset\sim Y_{g,t}^{\left(\left\{ x_{g,t,i}^{(1)} \right\}\right)}\) and \(\overset\sim Y_{g,t}^{\left(\left\{ x_{g,t,i}^{(0)} \right\}\right)}\) respectively.

The following would be instrumental for deciding between counterfactual scenarios:

However, this quantity is not observable because one cannot observe the potential outcome under both counterfactual scenarios. It is possible to observe, at most, only one counterfactual scenario! Instead, for any two counterfactual treatment scenarios, \(\{x^{(1)}_{g,t,i} \}\) and \(\{ x^{(0)}_{g,t,i} \}\), define incremental outcome as:

where \(\{z_{g,t,i}\}\) denotes the observed values for a set of control variables. This shorthand notation is used to indicate that the expectation is conditional upon the control random variables taking on these values. Using an MMM regression model and a carefully selected set of control variables, this conditional expectation is estimable. See Estimating incremental outcome using regression for more.

Typically, the sum is taken over \(g=1,\dots G\) and \(t=1,\dots T\). However, you can also define incremental outcome for any subset of these values.

Counterfactual scenarios used in Meridian

Although incremental_outcome provides enough flexibility to consider any counterfactual scenarios, Meridian by default uses a specific pair of counterfactual treatment scenarios. Since ROI, mROI, and Response Curves are built on default settings of incremental_outcome, this specific pair is important.

Meridian lets \(\left\{x^{(1)}_{g,t,i}\right\}\) be actual historical values and \(\left\{x^{(0)}_{g,t,i}\right\}\) be the same as the actual historical values, except with all values for one specific treatment variable set to its baseline value. For paid and organic media, the baseline values are zero. For non-media treatment variables, the baseline value can be set to the observed minimum value of the variable (default), the maximum, or a user-provided float.

For a given treatment variable \(q\), the incremental outcome is defined as:

\[\text{IncrementalOutcome}_q = \text{IncrementalOutcome} \left(\Bigl\{ x_{g,t,i} \Bigr\}, \Bigl\{ x_{g,t,i}^{(0,q)} \Bigr\} \right)\]

Where:

• \(\left\{ x_{g,t,i}\right\}\) are the observed treatment values
• \(\left\{ x_{g,t,i}^{(0,q)} \right\}\) denotes the observed treatment values for all treatments except treatment \(q\), which is set to its baseline value \(b_q\) everywhere. More specifically:
  • \(x_{g,t,q}^{(0,q)}=b_q\ \forall\ g,t\)
  • \(x_{g,t,i}^{(0,q)}=x_{g,t,i}\ \forall\ g,t,i \neq q\)

Note that the counterfactual media scenario (\(\left\{ x_{g,t,i}^{(0,q)} \right\}\)) may not actually be represented in the data. When this happens, extrapolation based on model assumptions is necessary to infer the counterfactual.

ROI

The ROI of a paid media channel \(q\) is defined as:

\[\text{ROI}_q = \dfrac{\text{IncrementalOutcome}_q}{\text{Cost}_q}\]

Where \(\text{Cost}_q\) is the cost of paid media channel \(q\) summed over all geos and time.

Note that the ROI definition uses a channel's total cost over a specific period as the denominator. The numerator is the incremental outcome accrued during that same period. This numerator includes lagged effects from ads that ran before the period but excludes future effects from ads that ran during the period. Over a long time window (for example, one year), this has a minor effect on the ROI estimate. However, for shorter periods, the effect can be more meaningful.

Response curves

Generalizing the incremental outcome definition, the response curve is defined for paid media channel \(q\) as a function which returns the incremental outcome as a function of the spend on channel \(q\):

\[\text{IncrementalOutcome}_q (\omega \cdot \text{Cost}_q) = \text{IncrementalOutcome} \left(\left\{ x^{(\omega,q)}_{g,t,i} \right\}, \left\{ x^{(0,q)}_{g,t,i} \right\}\right)\]

Where \(\left\{ x^{(\omega,q)}_{g,t,i} \right\}\) denotes the observed media values for all channels except channel \(q\), which is multiplied by a factor of \(\omega\) everywhere. More specifically:

• \(x^{(\omega,q)}_{g,t,q}=\omega \cdot x_{g,t,q}\ \forall\ g,t\)
• \(x^{(\omega,q)}_{g,t,i}=x_{g,t,i} \forall\ g,t,i \neq q\)

Marginal ROI (mROI)

The marginal ROI (mROI) of paid media channel \(q\) is defined as:

Where \(\delta\) is a small quantity, such as \(0.01\).

Note that the response curve and marginal ROI definitions implicitly assumes a constant cost per media unit that equals the historical average cost per media unit.