What is Incrementality Testing in Performance Marketing?

Imagine you're running a retargeting campaign. Your click-through rates are solid, your ROAS looks great, and conversions are rolling in. But here's the uncomfortable question most marketers never ask: Would those customers have converted anyway, even without seeing your ad?

This is the core problem that incrementality testing is designed to solve.

Let's understand how an actual incrementality test works, not in theory, but in practice. We'll use a direct-to-consumer skincare brand as our example. Call them GlowLab (imaginary brand).

The Situation

GlowLab is spending $80,000/month on Meta ads. Their Meta dashboard reports a 4.2x ROAS on their retargeting campaigns, people who visited the site but didn't purchase. The growth team wants to scale the budget. Before they do, they want to know: are these retargeting ads actually changing customer behaviour, or are they just collecting credit from people who were already going to buy?

Step 1: Define the Conversion Event and Baseline

GlowLab's primary conversion event is a completed purchase on their Shopify store. Their data team pulls 90 days of historical data and establishes a baseline conversion rate for their retargeting audience: roughly 3.1% of website visitors who don't purchase on the first visit end up buying within 14 days.

Step 2: Build the Test and Control Groups

Using their customer data platform, GlowLab randomly splits their retargeting audience into two buckets:

• Test group (80% of the audience): Served Meta retargeting ads as normal.

• Control group (20% of the audience): Suppressed from seeing any Meta retargeting ads for the duration of the test. This group either sees a blank space where the ad would have been, or is served a public service announcement — a "ghost ad", to absorb the impression without influencing purchase behavior.

The split is done at the user level, randomized by user ID to ensure it's statistically clean and the two groups are comparable across age, geography, device type, and historical purchase behavior.

Step 3: Run the Test

The test runs for 4 weeks, long enough to capture a full purchase cycle and reach statistical significance given GlowLab's audience size. During this period, the data team monitors for contamination (e.g., users switching between groups) but does not analyze results early. Peeking at results before significance is reached is one of the most common ways incrementality tests produce misleading data.

Step 4: Measure the Results

At the end of week four, the results come in (check table 4.1)

The incremental lift is the difference: 3.7% - 2.9% = 0.8 percentage points.

That means only 0.8% of the total retargeting audience converted because of the ads. The remaining 2.9% would have converted anyway.

Step 5: Calculate True Incremental ROAS

Of the 1,480 conversions in the test group, GlowLab estimates that roughly 78% (about 1,154) would have happened without any ads, based on the control group's rate. Only 326 conversions were genuinely incremental.

They recalculate ROAS using only those 326 incremental conversions. Average order value is $72. That's $23,472 in incremental revenue against $64,000 in ad spend for the test group.

Incremental ROAS: 0.37x.

Not 4.2x. 0.37x. The campaign was losing money in real terms.

Step 6: Act on the Data

This doesn't mean GlowLab kills the retargeting campaign entirely. But it does mean:

• They dramatically reduce retargeting spend and reallocate to prospecting campaigns, which a parallel incrementality test shows have a 2.8x incremental ROAS.

• They tighten retargeting windows (targeting only users who visited in the last 3 days rather than 30) to focus on higher-intent visitors where ads may have more genuine impact.

• They now have a repeatable testing framework to measure any channel before scaling it.

What Made This Test Work

A few things GlowLab got right that many teams don't:

• Clean group separation- no user appeared in both test and control.

• Sufficient holdout size- 20% gave them enough conversions in the control group to be statistically meaningful.

• Full-cycle duration- 4 weeks captured the real purchase window, not just last-click behavior.

• Custom infrastructure- this wasn't done inside Meta's dashboard. It was built on their own data stack, giving them neutral, platform-independent results.

This is why incrementality testing requires custom setup.

If you're ready to build a more accurate picture of your marketing performance, here's a high-level framework:

1. Define your key conversion events- purchases, sign-ups, qualified leads, etc.

2. Choose the channel or campaign to test- start with your highest-spend area.

3. Design your holdout- determine what percentage of your audience to withhold and for how long.

4. Ensure clean separation- test and control groups must be isolated to prevent contamination.

5. Run the test to statistical significance- don't end it early based on early trends.

6. Analyze and act- compare conversion rates, calculate true incremental lift, and use the data to inform budget decisions.
   

At Streamlyner, we build best incrementality testing tools for performance marketing teams. Every setup is tailored to your data, channels, and infrastructure, with a 100% money-back guarantee. If you're ready to find out what your ads are actually doing, check out our work.

For teams serious about measuring real impact, not just dashboard metrics, it's one of the most valuable investments in your measurement stack.
Get the best incrementality tools at Streamlyner.