The 2024 Guide to Incrementality

Have you or a loved one recently been inundated with messages about incrementality? Can’t figure out if this is the latest fad or a legitimate evolution in analytics? In today’s fast-paced and data-driven marketing landscape, staying ahead of the curve is essential for success. In a sea of buzzwords and trends, incrementality has emerged as a contender with significant implications for marketing strategy and performance measurement. In this article, we’ll cover what exactly this purported “silver bullet” is, and what marketers need to know. 

Section 1: What is incrementality? 

Incrementality refers to the measure of the additional (or incremental) lift that a marketing initiative provides compared to baseline expectations. In other words, incrementality seeks to answer the question of “what additional value did my marketing activities contribute beyond what would have occurred naturally?”

Incrementality helps answer this question by comparing the performance of those who were exposed to marketing campaigns (treatment group) with a similar cohort that did not see those same campaigns (control group). By measuring the difference in outcomes between the two, we can determine the additional value generated by the campaign. 

You’ll notice this is essentially describing the scientific method; like scientific experimentation, incrementality analysis begins with hypothesis testing, where marketers conduct randomized experiments to isolate the incremental impact of campaigns. This emphasis on empirical evidence is why incrementality is so revered: of all the popular analytical methodologies employed by marketers, incrementality is the only one that establishes a causal relationship as opposed to simply explaining the correlation between the independent and dependent variables. 

These experiments can take many forms, but they all generally involve some way of randomly selecting users into a treatment or control group. A geographic experiment, for example, would segment the country into similar pockets and then randomly assign them to a treatment and control group. This is also called a holdout test since we’re withholding marketing exposure from a portion of the target audience. 

Section 2: How is incrementality used?

Since we can rely on incrementality to establish a causal relationship, it’s often a great tool to validate results or justify marketing spend. Unlike other common marketing analytics tools that mostly conduct correlational analysis, this gives our data more credibility. The next time your finance department asks why they should keep spending dollars on a marketing initiative, you can simply refer to the results of your latest incrementality experiments with confidence. Without the marketing spend that you’re counting on, the company would be leaving money on the table. This really helps change the message from looking at marketing as an expense, and more as an investment. 

Incrementality is commonly used for budget allocation purposes because we can run various types of randomized experiments on our top channels and campaigns to test if those marketing dollars are truly driving new revenue instead of cannibalizing existing sales. Not sure if Facebook Display is actually driving positive results for your business? Design an incrementality test and find out. Repeat the process for your top channels and let those results help you decide how to divvy up your marketing spend. Tactically, incrementality is also well-suited for channels that your team doesn’t have much experience with. Let’s say your team wants to spend more on a new channel, but you’re not sure if additional spend will generate the necessary ROI. If we run a pilot and conduct incrementality analysis on the results, we can extrapolate and forecast what a scale up would look like.

Section 3: What are some challenges of incrementality? 

While incrementality testing is a powerful tool for measuring the true impact of marketing efforts, there are some challenges and limitations you should be aware of. 

• Since incrementality involves experiments, a big challenge is ensuring the sample size is sufficiently large enough so that our results are statistically significant. Small sample sizes can lead to low statistical power, making it difficult to draw meaningful conclusions from the results. 
• It’s impossible to remove all of the “noise” surrounding a marketing campaign so an omitted or hidden variable could dramatically skew your results. Did you hire more sales reps while the campaign was running? Did you increase more communication with the system so more sales occurred? When you randomly assigned treatment and control groups, were there hidden mechanisms that prevented true randomization such as inherent biases in the selection process? Remember that incrementality attempts to show causation, which means your chances of getting a bad read increase if there are important omitted variables. Incrementality also struggles to account for spillover effects (marketing efforts often impact individuals who were not directly exposed to the campaign, through word of mouth and other means) which are particularly important in digital marketing where these cross-channel interactions are more common. 
• Incrementality is not a low maintenance, set it and forget it, type of tool. To get the most out of this technique requires a skilled marketing team that has a larger strategy and framework (in the form of a learning agenda or testing plan) that will guide what to test, when to test it, and most importantly, how to interpret those results and make actionable decisions off those insights. 
• Note that incrementality experiments generally run for weeks not days, during which we cannot make changes to our campaigns because it will impact results. That means no swapping out of creative, A/B testing, or introducing any other variables while experiments are running. This is not ideal for larger channels because it requires us to disrupt our business as usual operations to accommodate a statistical technique. 

Section 4: Incrementality vs. other popular analytics methods 

Incrementality is often grouped together with other popular measurement methodologies including multi-touch attribution (MTA) and marketing/ media mix modeling (MMM). While all 3 can help marketers understand the effectiveness and impact of their efforts, they evolved for different reasons and thus are best suited for different things. 

Before the digital advertising schism moved us away from traditional advertising channels like linear TV, MMM was the primary form of media measurement because it doesn’t rely on granular data. MMM excels at capturing older channels and the longer-term impact on revenue by leveraging historical data (both media and macroeconomic data) to predict future sales based on past behavior. However, MMM is very expensive to spin up and often takes months to deploy with a reporting lag after each quarter to update the model, so other methodologies sprang up to make up for these deficiencies. 

MTA came next with the rise of digital media, along with detailed user tracking which gave marketers more granularity in terms of insights. By leveraging this richer user data to track customer journeys, MTA was far better adapted for the complex and cross-channel landscape and is the most bottoms-up approach as it collects data at the individual level and then aggregates this to create a model. However, with the rise of consumer privacy regulations, MTA may become less effective in the future if user-level tracking goes completely away (still looks to be sometime in the future given Google recently delayed cookie deprecation again) so marketers began to turn to incrementality experimentation. 

You’ll notice that none of these methodologies are silver bullets; each have their own distinct advantages and disadvantages. The truth is that the best marketing teams leverage some combination (or all) of these 3 to guide their decision making because this can make up for individual weaknesses. Incrementality is simply the latest in a long line of measurement methods that marketers are using to try and gain an edge. Although the promise of finding causal relationships is tantalizing, the challenge of setting up experiments correctly keeps many companies from effectively using incrementality. If your audiences somehow overlap, your segment sizes are off, or a number of other possibilities somehow skew your results, you’re using incorrect data to make very impactful decisions (potentially for the worse). 

If you want to learn more about the MTA, MMM and how they compare to incrementality, we wrote this awesome deep dive that goes into much more depth on what MTA and MMM are, and also provides a detailed comparison of the three. Click here to check that out.