Incrementality Testing: Measuring What Your Ads Actually Cause

Incrementality testing is a method for measuring the true causal effect of advertising on conversions. It answers the question attribution cannot: "If we hadn't run this campaign, how many of these conversions would have happened anyway?"

The core setup is a holdout experiment. You split your target audience into two groups: a test group that receives ads and a holdout group that doesn't. After the test period, you compare conversion rates. The difference between the two groups is the incremental lift from advertising.

Why Attribution Isn't Enough

Attribution tells you which touchpoints received credit for conversions. It doesn't tell you whether those touchpoints caused the conversions.

Consider a user who was going to download your app because they saw it on a friend's phone. Before they do, they're also served a retargeting ad. When they download, the retargeting campaign gets credit (last-touch) or shared credit (data-driven). But the ad didn't cause the download — it just appeared before an intent that already existed.

Last-touch attribution makes retargeting look more effective than it is because retargeting reaches users who are already inclined to convert. These users convert at high rates regardless of the ad, but the ad claims credit because it was the most recent touchpoint before conversion.

The same problem affects prospecting campaigns in the opposite direction. A user who saw a top-of-funnel awareness ad in January, converted organically in March, and then clicked a brand search ad before purchasing in May will credit the brand search campaign — not the January awareness campaign that started the relationship.

Multi-touch attribution models try to distribute credit more fairly, but they're still correlation models. None of them can tell you whether the ads caused conversions.

Incrementality testing is the only method that can establish causation.

How Holdout Tests Work

The basic structure: randomly assign users to test and holdout groups before running the campaign. Run ads only to the test group. After the campaign window, compare conversion rates between the groups.

The key requirement is random assignment before exposure. If users self-select into groups (for example, holdout is defined as users who didn't see the ad because of targeting mismatches), the comparison is contaminated. Random pre-assignment ensures the groups are comparable at baseline.

Most ad platforms support holdout experiments natively:

• Meta: Brand Lift Studies and Conversion Lift Studies
• Google: Conversion Lift with Google Ads (requires minimum budget and audience size)
• AppsFlyer: Incrementality measurement (hold-out groups at campaign or ad set level)
• Adjust: Incrementality module

Running holdout tests through your MMP gives you cleaner cross-channel measurement, because the holdout is defined at the user level regardless of which platform they're exposed to.

Interpreting the Results

The main output is incremental conversions: the number of additional conversions that happened because of the ads, versus what would have happened without them.

Incremental ROAS (iROAS) is often more actionable than platform-reported ROAS. It's calculated as: total revenue from incremental conversions / total ad spend.

If your campaign drove 500 total attributed conversions but the holdout test suggests only 200 were incremental (the other 300 would have happened anyway), your iROAS is roughly 40% of what platform reporting shows. A campaign that looked like 5x ROAS in the platform is actually running at around 2x incremental ROAS.

What's a good iROAS? This depends on your margin structure (same framework as ROAS). The question is whether the iROAS covers your incremental cost of goods and contributes to overhead. If iROAS is below your minimum viable threshold, the campaign is net-negative even if platform reporting looks positive.

What Incrementality Tests Reveal

The most common findings from incrementality testing that surprise growth teams:

Retargeting has less lift than it appears. Users in retargeting pools are pre-qualified by behavioral history — they've already expressed intent. Many of them will convert without seeing additional ads. The holdout group in a retargeting test often converts at 60-80% of the rate of the exposed group, meaning the actual incremental lift from retargeting is 20-40%, not 100%.

Brand search has near-zero incrementality for strong brands. Users searching your brand name were going to your site anyway. Brand search ads mostly intercept organic visits, inflating paid conversion numbers without adding real lift. For most established brands, brand search ad spend has very high attribution but very low incrementality.

Prospecting campaigns have more lift than they appear. Because prospecting uses top-of-funnel touchpoints that rarely get last-click credit, its actual incrementality is often higher than attributed conversions show. The January awareness campaign that eventually led to a May purchase shows up as zero conversions in last-touch attribution but represents real incremental lift.

When to Run Incrementality Tests

Run incrementality testing before making significant budget decisions. If you're considering scaling a channel 3x because of strong platform ROAS, an incrementality test first tells you whether you'd actually be scaling profitable growth or scaling credit-claiming.

High-priority candidates for incrementality testing: retargeting campaigns (highest risk of ROAS inflation), brand search (often zero incrementality), and any channel where platform-reported ROAS is suspiciously high relative to business outcomes.

AIMA tracks incrementality data alongside attribution data, which gives a more accurate view of which campaigns are generating real growth versus claiming credit from organic intent.

How to Act on Incrementality Results

The test gives you a number: incremental lift percentage and iROAS. The harder question is what to do with it.

If iROAS is above your profitable threshold, scale the channel. The lift is real and there is room to grow.

If iROAS is below your breakeven threshold, the math is clear: the channel is losing money on a net basis, because conversions that would have happened anyway are being counted as campaign wins. Reducing spend or pausing the campaign is the right call unless there is a strategic reason, such as competitive coverage or brand presence, to accept a money-losing channel temporarily.

The harder case is iROAS between breakeven and your target margin. The channel is generating incremental value but not efficiently. Several levers can improve it.

Better audience segmentation. The broadest retargeting audience (all non-converters) has the lowest incrementality because it includes many users who would have converted without any ads. Narrower high-intent segments, such as trial completers who did not subscribe, users who reached the paywall multiple times, or users who lapsed within the first 72 hours, have higher incrementality because they need more of a nudge to convert and would not have done so without one.

Changing the creative message. A conversion-focused message on a retargeting audience, one that directly addresses the specific friction point (price, feature uncertainty, competition), tends to generate higher incremental lift than a general brand awareness creative shown to the same group.

Attribution window adjustment. If you are optimizing on a 30-day click window but most incremental conversions happen within 7 days of ad exposure, a tighter window gives the algorithm more accurate signal and may improve both iROAS and delivery efficiency.

Incrementality tests should run quarterly on your largest channels, not as a one-time audit. Channel lift shifts over time as audiences saturate, creative refreshes happen, and competitive dynamics change. A channel that had strong incrementality six months ago may look different today.