Calculate marketing incrementality using holdout tests. Compare total revenue with holdout-projected revenue to measure true campaign lift and ROI.
A holdout test withholds marketing from a randomly selected portion of your audience or geographic area, then compares outcomes between the exposed and holdout groups. The revenue difference, adjusted for group sizes, reveals the true incremental impact of your marketing spend.
This calculator takes the total revenue generated, the holdout group's projected revenue (extrapolated to the full audience), and the campaign spend to compute incrementality, incremental revenue, and incremental ROI. It's the most reliable method for answering: "What would have happened if we hadn't run this campaign?"
Holdout tests are essential for validating attribution models, optimizing media mix, and preventing wasteful spend on campaigns that don't drive true incremental value. Every sophisticated marketing organization runs regular holdout tests.
Precise measurement of this value supports data-driven marketing decisions and helps teams demonstrate clear return on investment to stakeholders and executive leadership. Quantifying this parameter enables systematic comparison across campaigns, channels, and time periods, revealing opportunities for optimization that drive sustainable business growth.
Holdout tests answer the fundamental question: does this marketing actually work? By measuring what happens when you stop marketing to a subset, you get the clearest picture of incremental value and can calculate true ROI on your marketing investment. Regular monitoring of this value helps marketing teams detect shifts in audience behavior early and adapt strategies before competitive advantages are lost in the marketplace.
Holdout Projected Revenue = Holdout Revenue / Holdout % × 100% Incremental Revenue = Total Revenue − Holdout Projected Revenue Incrementality = Incremental Revenue / Total Revenue × 100 iROI = (Incremental Revenue − Spend) / Spend × 100
Result: Incremental Revenue: $140,000 | iROI: 75%
Holdout (10% of audience) generated $36,000. Projected full-audience without marketing: $36,000 / 10% = $360,000. Incremental revenue: $500,000 − $360,000 = $140,000. iROI: ($140,000 − $80,000) / $80,000 × 100 = 75%.
Holdout tests provide the most rigorous answer to the fundamental marketing question: what is the true incremental value of our spend? Unlike attribution models that redistribute credit for conversions that may have happened anyway, holdout tests reveal the counterfactual by actually observing what happens without marketing.
The key to a reliable holdout test is proper randomization. Whether at the user level or geographic level, the holdout group must be statistically equivalent to the exposed group in all relevant dimensions (demographics, purchase history, activity level). Any systematic differences between groups will bias results.
Avoid ending tests early when results look good (or bad) — this creates statistical bias. Don't contaminate the holdout group with marketing from other channels. Account for the full conversion lag before concluding. And be transparent about the revenue sacrifice: holdout testing has a short-term cost for long-term optimization gains.
A holdout test withholds marketing activity from a randomly selected portion of your audience (the holdout/control group). By comparing outcomes between the exposed group and the holdout, you can measure the true incremental impact of your marketing investments.
Typically 10–20% of your audience. Larger holdouts provide more statistical power but reduce campaign reach. The optimal size depends on your conversion volume, expected lift, and desired confidence level.
An A/B test compares two variations of a marketing element (e.g., two ad creatives). A holdout test compares marketing exposure vs. no marketing. Holdout tests measure whether marketing works at all; A/B tests optimize which version works best.
Incrementality rate is the percentage of total conversions or revenue that would not have occurred without the marketing campaign. A 30% incrementality rate means 30% of conversions were truly driven by marketing and 70% would have happened anyway.
In geographic holdouts, you select matched pairs of similar markets (cities, DMAs). One in each pair receives the campaign, the other doesn't. Comparing performance between exposed and holdout markets reveals the campaign's incremental impact.
Yes, holding out audience means some potential customers won't see your campaign. However, the revenue impact is typically small (10–20% holdout at most), and the insights gained help optimize 80–90% of your spend for much greater long-term returns.