Module 4 — Campaigns and Experiments · Lesson 4.2
Running Experiments That Mean Something
A/B tests, sample size, why peeking lies, and what a small team can honestly conclude
What you'll learn
- Explain why a randomized controlled experiment establishes causation where a before-and-after comparison cannot
- Choose a single overall evaluation criterion before the test starts
- Explain the peeking problem and commit to a sample size or a pre-set stopping rule
- Judge which questions are testable at your volume and which are not
Every marketing team has opinions about subject lines, offers, page layouts and send times. Experiments replace opinion with evidence — but only when they are designed and read correctly. A badly run test is worse than none, because it produces a confident wrong answer with a percentage attached, and the team then defends it with the authority of 'the data'.
Why controlled experiments
Ron Kohavi, Roger Longbotham, Dan Sommerfield and Randal Henne's 2009 survey Controlled Experiments on the Web: Survey and Practical Guide — written from running experimentation at Microsoft and elsewhere — describes controlled experiments (A/B tests, split tests, multivariate tests) as embodying the best scientific design for establishing a causal relationship between changes and their influence on user-observable behaviour. Randomly assigning people to versions means that, on average, the groups differ only in the change you made, so a difference in outcomes can be attributed to it. A before-and-after comparison cannot do that: anything else that changed between the two periods — season, a competitor's launch, a news cycle — is mixed into the result.
The same authors argue that organizations learn most when they listen to customers rather than the Highest Paid Person's Opinion, which they abbreviate HiPPO. Kohavi and Stefan Thomke's 2017 Harvard Business Review article gives the canonical illustration: in 2012 a Microsoft employee proposed changing how Bing displayed ad headlines; the idea was rated low priority and sat for more than six months until an engineer ran a simple A/B test. Revenue rose so quickly it triggered an alarm, and the analysis showed a 12% increase — estimated at over $100 million a year in the US — without hurting user-experience metrics. Nobody's intuition had rated it highly. That is the point of testing.
One criterion, decided first
Kohavi and colleagues define the Overall Evaluation Criterion (OEC) as a quantitative measure of the experiment's objective — what statisticians call the response or dependent variable. Choose it before the test starts, and choose one. If you measure ten metrics and declare victory on whichever moved, you have run ten tests and reported the luckiest.
For marketing, the OEC should be as close to the outcome you care about as your volume allows. For a cold outreach sequence that is usually replies or meetings, not opens. (Kavanah does not track opens at all, so for cadence tests the choice is made for you.) For a landing page it is the conversion you actually want, not time on page.
The survey also describes the A/A test: splitting users into two groups that see exactly the same thing. It is a cheap way to check that your system and your statistics behave — an A/A test that 'finds' a winner is telling you something is broken.
Peeking, and fixing the sample size
Evan Miller's 2010 essay How Not To Run an A/B Test explains the most common error. If you check a running test repeatedly and stop as soon as it shows a significant difference, the reported significance level becomes meaningless: repeated significance testing always increases the rate of false positives. In his example, stopping at the first significant result when the change actually does nothing produced a false-positive rate of 26.1% — more than five times the nominal 5%.
His remedy is simple: decide on a sample size in advance and wait until the experiment is over. Committing to the sample size completely removes the problem. If you genuinely need the option to stop early, use a method designed for it — sequential or Bayesian designs — rather than an ordinary significance test checked every morning.
The arithmetic of sample size is sobering for a small team. Detecting a small difference in a low base rate takes thousands of people per arm. Using the standard two-proportion approximation at 5% significance and 80% power, telling a 3% reply rate from 6% needs roughly 750 sends per arm; telling 3% from 3.5% needs nearly 20,000 per arm. Work it through for your own numbers — it is the fastest cure for testing subject-line wording. So test big differences — a different offer, a different job, a different sender — rather than a word in the subject line, and accept that many questions are not answerable by experiment at your volume. Use an online sample-size calculator before every test and write the number down.
What Kavanah's cadence experiments do
Kavanah supports A/B testing on outbound cadences: up to four arms (A to D) varying the subject line and the sender. A test can be linked (subject and sender change together, one comparison) or factorial (subject and sender measured as separate factors). Results report sends, replies and reply rate per arm, and per factor the replies, positive replies and meetings. 'Positive reply' is a stated proxy — a meeting, or a reply from someone who has not opted out and whose deal status is not a decline — not a sentiment classifier.
Automatic winner selection is opt-in and admin-only. It uses a Bayesian probability-to-be-best calculation, and it will not judge a winner until every arm has reached a minimum number of contacts; below that it reports 'collecting' and names the leader as 'not a result yet'. Those two design choices — a Bayesian rule and a minimum per arm — are the guards against the peeking problem above. They are not a licence to set the minimum low: set it to the sample size your calculation says you need.
Run one honest cadence test
- 1
Write the hypothesis and sample size first
On /projects/tasks, create the test task with the hypothesis, the single criterion (replies or meetings), the arms, and the per-arm sample size from a calculator.
- 2
Configure the variant test on a cadence
In /clients, open the Deal Room's Manage cadences and set up the experiment on a cadence — linked if you are testing one combined change, factorial if you want subject and sender read separately. You can also ask the agent on /ai-agent to suggest variants; you choose which to run.
- 3
Ask the agent for the cadence's variant stats once every arm has reached your sample size. If you enable automatic winner selection, set its minimum per arm to that number, not lower.
- 4
Record the result, including null results
On /knowledge, log what was tested, the numbers and the conclusion. A test that found no difference is still knowledge.
Experiment discipline
- Tests with a pre-registered sample size
- Share of tests whose hypothesis, criterion and sample size were written down before the first send.
- Healthy signal: All of them.
- Reply rate per arm
- Replies divided by sends per arm, from Kavanah's cadence variant stats.
- Healthy signal: Read only at the planned sample size; differences smaller than your test was powered to detect are not results.
- Decisions changed by tests
- Number of times in a quarter a test result changed what the team did.
- Healthy signal: At least one. Tests that never change a decision are being chosen too timidly or ignored.
Key takeaways
- ·Randomized experiments establish cause; before-and-after comparisons cannot.
- ·Pick one overall evaluation criterion before the test, as close to the real outcome as volume allows.
- ·Peeking and stopping on significance inflates false positives; fix the sample size or use a method built for early stopping.
- ·At small volume, test big differences — and accept that some questions cannot be answered by experiment.
Experiments improve individual campaigns. Most of marketing, though, is not campaigns but rhythm — the newsletter every fortnight, the refresh every quarter. The next lesson treats the content calendar as recurring work.
Sources
- 1.Controlled Experiments on the Web: Survey and Practical Guide
Ron Kohavi, Roger Longbotham, Dan Sommerfield, Randal M. Henne · Data Mining and Knowledge Discovery 18(1), 140–181 · 2009
Controlled experiments as the best design for establishing causality; listen to customers, not the HiPPO; definitions of the OEC and the A/A test; statistical power and sample size.
- 2.The Surprising Power of Online Experiments
Ron Kohavi and Stefan Thomke · Harvard Business Review, September–October 2017 · 2017
The Bing ad-headline test: a deprioritized idea that lifted revenue 12% (estimated over $100M a year in the US) without hurting user-experience metrics.
- 3.How Not To Run an A/B Test
Evan Miller · evanmiller.org · 2010
Repeated significance testing inflates false positives (26.1% vs a nominal 5% in his example); fix the sample size in advance or use sequential/Bayesian methods.