Turning Incrementality Tests Into Action That Makes You Money
Ecommerce Playbook · with Olivia Kory & George Davis · February 26, 2026 · 90 min
Summary
This episode demystifies incrementality testing, offering e-commerce operators actionable strategies to translate test results into profitable budget decisions. It addresses common pitfalls like conflicting channel data and unreliable platform ROAS, providing a framework for optimizing spend and understanding true incremental impact beyond vanity metrics. Learn how to operationalize incrementality for real-world growth.
Key takeaways
Implement holdout groups in incrementality tests for accurate measurement, as 'spend up/spend down' tactics are insufficient for reliable data.
Utilize Incremental Marginal Return (IMR) over iROAS to compare channel performance more intuitively and make profit-driven budget allocations.
Focus on overall profit thresholds and an Incremental Marginal Return (IMR) framework for budget decisions instead of getting sidetracked by channel-versus-channel debates.
If iROAS is consistently below 1.0, don't automatically cut spend; instead, analyze creative, account structure, exclusions, and explore distribution expansion (e.g., Amazon, retail) to improve incremental efficiency.
Recognize the "replication problem" in incrementality tests; results can change month-to-month even with clean tests. Plan for iterative testing and agile budget adjustments based on evolving data.
Incrementality tests are “in”… but the real problem is what you do after the read.In this episode, Taylor sits down with Olivia Kory (Chief Strategy Officer at Haus) and George Davis (CMO at Cozy Earth) to unpack the messiest part of modern measurement: operationalizing incrementality when results swing, channels conflict, and “platform ROAS” can’t be trusted.If you’ve ever asked:“Our holdout came back way lower than Meta… now what?”“Why don’t test results replicate month-to-month?”“How do I actually use an incrementality factor in real budget decisions?”“If everything is under 1.0 iROAS… should we cut spend or keep investing?”…this one is for you.What we coverWhy incrementality requires a holdout (and why “spend up / spend down” isn’t enough)The replication problem: why results change even with “clean” testsThe gap between measurement and optimization (platforms optimize for attribution, not incrementality)How operators use incrementality factors without letting them become a blunt instrumentWhy channel vs. channel is often the wrong fight (and why profit thresholds matter more)iROAS → IMR (Incremental Marginal Return): a more intuitive way to compare performanceBudget cadence: daily realities vs monthly allocation decisionsLong-term effects, “adstock” claims, and why post-treatment windows matterPractical levers that can improve results: creative, account structure, exclusions, distribution expansion (Amazon/retail)Got a weird incrementality result? Drop it in the comments. We’ll let you know what we’d do next.Read this next: CTC Core Methodology Series: Marketing Measurement - https://bit.ly/4tW2JwFShow Notes:Axon i
What does this episode say about paid acquisition?
Implement holdout groups in incrementality tests for accurate measurement, as 'spend up/spend down' tactics are insufficient for reliable data.
What does this episode say about analytics & attribution?
Utilize Incremental Marginal Return (IMR) over iROAS to compare channel performance more intuitively and make profit-driven budget allocations.
What does this episode say about dtc strategy?
Focus on overall profit thresholds and an Incremental Marginal Return (IMR) framework for budget decisions instead of getting sidetracked by channel-versus-channel debates.
What does this episode say about paid acquisition?
If iROAS is consistently below 1.0, don't automatically cut spend; instead, analyze creative, account structure, exclusions, and explore distribution expansion (e.g., Amazon, retail) to improve incremental efficiency.
What does this episode say about paid acquisition?
Recognize the "replication problem" in incrementality tests; results can change month-to-month even with clean tests. Plan for iterative testing and agile budget adjustments based on evolving data.
