Module 5 — Measurement and Operations · Lesson 5.3
Attribution and Its Limits: Measuring Incrementality
Why attribution reports describe journeys rather than causes, and what a small team can do instead
What you'll learn
- Distinguish attribution (who touched the journey) from incrementality (what the marketing caused)
- Explain what large field experiments found when they compared observational methods with randomized tests
- Design a simple holdout or geographic test for one channel
- Combine self-reported source, experiments and judgement into a proportionate measurement practice
Every marketing dashboard has an attribution chart: last-click, first-click, linear, time-decay, data-driven. They look rigorous. They answer a question — which touchpoints appeared in the journeys of people who bought — that is different from the one that matters for budget: which of those touchpoints made people buy who otherwise would not have. The gap between those two questions is where most marketing budget is misallocated.
What the experiments found
Brett Gordon and Florian Zettelmeyer of Kellogg, with Neha Bhargava and Dan Chapsky of Facebook, used 12 US advertising lift studies at Facebook — 435 million user-study observations and 1.4 billion impressions — to compare randomized controlled trials with the observational methods common in industry: comparing exposed with unexposed users, matching, model-based adjustments, synthetic matched-market tests and before-after comparisons. Their 2016 white paper (published in Marketing Science in 2019) found that observational methods often fail to produce the same results as true experiments, even after conditioning on thousands of behavioural variables and using non-linear models, and concluded that common industry approaches fail to measure accurately the true effect of ads.
The reason is selection. Ads are targeted at people likely to buy, and people likely to buy are more likely to see and click ads. An exposed group therefore converts more than an unexposed group even when the ad does nothing. Attribution models built on the same journey data inherit the same bias.
The eBay experiments by Blake, Nosko and Tadelis (Lessons 3.1 and 4.4) found the same pattern from the other side: switching paid search off in some markets showed returns that were a fraction of non-experimental estimates, with brand-keyword ads showing no measurable short-term benefit. And Lewis and Rao's analysis of 25 retail ad experiments showed that even randomized tests produce very wide confidence intervals on return on investment because sales are so noisy relative to ad cost.
Incrementality, defined
Incrementality is the difference between what happened with the marketing and what would have happened without it. It can only be estimated by comparing a group that received the marketing with a comparable group that did not, where the difference between the groups was decided by chance or design rather than by who was likely to buy.
Three practical designs. A holdout: randomly withhold a campaign — an email, a cadence, a retargeting audience — from a slice of the eligible audience and compare outcomes. A geographic test: switch a channel on in some regions and off in comparable others, as eBay did with search. A time-based switch-off: pause a channel for a defined period and watch whether the outcome falls; weaker, because other things change over time, but often the only option for a small company.
Tools have improved. Google released Meridian, an open-source marketing mix model, and in October 2026 launched Meridian GeoX, an open-source geo-experimentation library intended to let marketers run cross-publisher geo experiments and use them to calibrate marketing mix models. Google is, of course, an interested party in how advertising is measured; the value of open-source tools is that the method is inspectable, not that the vendor is neutral.
Proportionate measurement for a small team
Most small companies do not have the volume for precise incrementality measurement; Lewis and Rao's finding about noise applies with greater force at small scale. That is not a reason to fall back on attribution charts. It is a reason to combine several imperfect signals, each with known weaknesses.
Ask every new lead how they heard about you, and record the answer verbatim; self-reported source captures word of mouth, podcasts and communities that no tracking link sees, though people misremember. Run a holdout whenever a channel is cheap to withhold — an email to customers, a nurture sequence — and expect only large effects to be detectable. Pause expensive channels deliberately, one at a time, for a planned period, and write down beforehand what drop would convince you the channel was working. And keep brand effects in view: the IPA and Ehrenberg-Bass work in Lesson 1.1 says much of marketing's value is long-term and will not show up in any short test, so the absence of a short-term effect is not proof that brand work is wasted.
Above all, separate the question 'which channels appear in buyers' journeys?' (attribution, useful for understanding) from 'what should we spend more on?' (incrementality, the only valid basis for budget).
What Kavanah attributes, and what you add
- 1
Know what Kavanah's attribution means
In /clients, Kavanah matches synced email replies and calendar meetings to contacts, deals and the cadence step that preceded them — so a reply stops the cadence and shows on the deal. That is engagement tracking, not marketing-source attribution, and it makes no causal claim.
- 2
Record self-reported source on every new deal
Tag each new contact with the source they gave in their own words. Ask the agent on /ai-agent to summarize the source tags across the quarter's new deals.
- 3
When enrolling a list in a cadence, deliberately leave a random slice unenrolled and record which contacts were held out. Compare meetings booked between the groups after a fixed period. It is manual, small and honest.
- 4
Plan channel pauses as experiments
On /projects/tasks, schedule each pause with its start, end and the pre-registered drop that would convince you.
Measurement practice
- Self-reported source coverage
- Share of new deals with a recorded, verbatim source.
- Healthy signal: Nearly all of them.
- Channels with an incrementality test in the last year
- Count of channels above a spend threshold you set that have had a holdout, geo test or planned pause.
- Healthy signal: Every significant paid channel at least once a year — our recommendation, not an industry standard.
- Holdout lift
- Outcome rate in the treated group minus the holdout group, with the sample sizes reported alongside.
- Healthy signal: Read with its uncertainty; at small samples only large differences mean anything.
Key takeaways
- ·Attribution describes who touched the journey; incrementality measures what the marketing caused.
- ·Large field experiments found observational ad measurement often fails to match randomized results because of selection.
- ·Holdouts, geographic tests and planned pauses estimate incrementality; open-source tools now exist for the latter.
- ·At small scale, combine self-reported source, simple experiments and judgement — and do not mistake missing short-term effects for wasted brand work.
The last lesson brings the course together around a question every marketing team now faces: an AI agent that can research, draft and send. How do you get the leverage without losing the trust?
Sources
- 1.A Comparison of Approaches to Advertising Measurement: Evidence from Big Field Experiments at Facebook (white paper)
Brett Gordon, Florian Zettelmeyer, Neha Bhargava, Dan Chapsky · Kellogg School of Management (published in Marketing Science 38(2), 2019) · 2016
12 lift studies, 435M user-study observations: observational methods often fail to match RCT results even with thousands of variables; common industry measurement fails to measure the true effect of ads.
- 2.Consumer Heterogeneity and Paid Search Effectiveness: A Large Scale Field Experiment
Thomas Blake, Chris Nosko, Steven Tadelis · NBER Working Paper 20171 (published in Econometrica, 2015) · 2014
Geographic switch-off of paid search at eBay: returns a fraction of non-experimental estimates; brand-keyword ads show no measurable short-term benefit.
- 3.The Unfavorable Economics of Measuring the Returns to Advertising
Randall A. Lewis and Justin M. Rao · Quarterly Journal of Economics 130(4), 1941–1973 · 2015
Even randomized ad experiments yield ROI confidence intervals over 100 percentage points wide; selection bias undermines observational methods.
- 4.Meridian GeoX: Google's new open-source geo incrementality solution
Google · business.google.com (October 9, 2026) · 2026
Open-source geo-experiment library for incrementality, designed to calibrate Meridian marketing mix models; vendor announcement.