Advantage+ reports your best ROAS and often your least incremental revenue. The campaign that looks best in the dashboard may be taking credit for buyers who were already reaching for their wallets. Scale it without checking, and you pour budget into harvesting demand you would have captured anyway.
An incrementality test separates real lift from attributed credit. One group sees ads while a comparable group does not. The difference tells you what the advertising actually caused, not just what the platform counted.
The short version: AI-automated campaigns can claim sales from people already likely to buy. A controlled holdout estimates the additional revenue the ads actually caused. Run one before any major budget change, and repeat when spending, channels, or demand shifts.
That test is part of how our Paid Media work checks reported performance against business outcomes.
What Is Incrementality Testing?
Incrementality testing estimates the revenue your ads caused beyond what would have happened without them. One group sees ads while a comparable control group does not. The difference in outcomes is incremental lift, subject to the test’s confidence range.
That distinction can change a budget decision. Standard ROAS follows the platform’s attribution settings, while a holdout estimates what happened because the ads ran. Common Thread Collective’s client analysis put median incremental ROAS for Google branded search at 0.27x in its sample. That finding should not be treated as a universal branded-search benchmark.
Google’s Conversion Lift documentation distinguishes attributed conversions from causal, incremental conversions. At the P&L level, the test asks: how much revenue would likely disappear if this campaign stopped?
Why Do AI-Automated Campaigns Inflate Reported ROAS?
Platform optimization and business measurement answer different questions. The system seeks attributed conversions or value. You need to know how much demand the campaign added beyond the control baseline. AI campaigns can widen that gap.
In Haus’s incrementality dataset, 58% of brands had higher incremental ROI from manual campaigns than from Advantage+. The difference between reported and incremental performance was about 12 percentage points in that sample.
Advantage+ may still win your own test. The finding is a reason to check, not to switch it off. Reported ROAS and incremental ROAS measure different things, so one cannot stand in for the other.
| Metric | Reported ROAS | Incremental ROAS |
|---|---|---|
| What it counts | Every conversion the platform attributes | Only sales the ad actually caused |
| Who reports it | The ad platform | The experiment design and analysis |
| Main limitation | Depends on attribution rules | Depends on power and control quality |
| Effect of AI automation | Can favor attributed demand | Estimates lift against a baseline |
| Use it to | Optimize inside a campaign | Decide budget between campaigns |
What Are the Main Incrementality Test Methods?
Three methods matter for most advertisers: geo holdouts, platform conversion lift, and public service announcement (PSA) tests. Each creates a control in a different way. The right method depends on geography, conversion volume, platform access, and the decision you need to make.
A geo holdout compares revenue across test and control markets. Conversion lift uses a platform-managed holdout of users inside Meta or Google. A PSA test shows the control group a charity ad to separate the effect of ad exposure from audience quality.
| Method | Best for | Main limitation |
|---|---|---|
| Geo holdout | Automated campaigns, cross-channel truth | Needs national or multi-region spend |
| Conversion lift | Single-platform reads, faster setup | Graded by the platform being tested |
| PSA test | Isolating exposure from audience quality | Costs budget on the control ad |
For cross-channel or publisher-independent questions, we often lean on geo holdouts. Google’s geo-experiment research describes how randomly assigned regions can measure advertising effects. A platform lift study can be better when user-level randomization and sufficient power are available.
Not sure how much of your ROAS is real?
We build incrementality into how we run paid media, so budget decisions rest on caused revenue, not platform claims.
Book a Free Strategy CallHow Do You Run a Geo Holdout Test on an AI Campaign?
Assign comparable markets to test and control groups, with no campaign ads shown in the control markets. Keep other conditions stable. Once the test meets its planned duration and statistical standard, compare outcomes to estimate the incremental result.
Careful setup makes the number trustworthy. Match your test and control markets on size, seasonality, and baseline demand before you start. A common failure is comparing your best market against your weakest and calling the difference lift.
Write the exclusion rules and decision threshold before launch. Otherwise, a noisy result becomes whichever story the team prefers.
Document planned spend, conversion lag, and outside promotions too. A pricing change or major email campaign can break market comparability. Flag those events before analysis and explain whether the test can still support a decision.
Run these steps in order:
- Use enough markets per group for the planned statistical power. Our starting screen is often 10 or more.
- Match the groups on historical revenue and trend, not just population.
- Establish a clean pre-period baseline before any change.
- Hold the control dark for the full planned window. Our tests often run 4 to 6 weeks.
- Compare incremental revenue against incremental spend to get true incremental ROAS.
Use the gap to adjust the budget. A campaign that reports 8x but tests at 2.8x may still be profitable, though it can support much less spending than the dashboard suggests. We apply the same discipline during a full paid media audit.
How Often Should You Re-Test Incrementality?
Re-test around major decisions: a new campaign type, a large budget shift, or a significant change in demand. Incrementality is not fixed. It moves with auction competition, creative, spend, and demand mix. Google says most advertisers run one or two Conversion Lift studies per year. Higher-spend teams may justify quarterly testing.
A campaign that measured 3.5x in Q1 can become less incremental as spending grows and the next buyers add less value. Pair that result with the unit economics in our CAC benchmarks work. Together, they show where the next dollar belongs.
The paradox from the top of this article runs through every budget review. The campaign with the best reported ROAS is often the one harvesting demand you would have captured anyway. An incrementality test tells you which campaigns create demand and which just claim it. Fund the creators, check the harvesters, and re-test before the numbers go stale.
Before your next large budget change, we can help you decide whether a holdout would answer the question that matters. Bring your campaign reports, conversion volume, and planned spend change. Book a Free Strategy Call to discuss a test design and the evidence you would need to act on it.
Like this? Get the next one.
Short emails. New posts as they ship.
