Skip to content

🚀 Join the team! We’re looking for a new Digital Marketing Manager. View the Role.

A/B Testing Emails: How to Run Tests That Actually Teach You Something

Why most email A/B tests teach you nothing

A/B testing helps you learn what your audience responds to, so you can make better decisions in future campaigns. That starts with a clear hypothesis: what do you expect to happen, and why?

Without one, even a winning result can be hard to explain. Too small a sample can make chance look like a meaningful difference. And if you only pay attention to results that support your expectations, you miss evidence that could improve your approach. This guide explains how to avoid all three problems.

Of 'winning' tests that fail to replicate when re-run on a larger sample
~50%
Minimum recipients per variant most tools need before a result means much
1,000+

Start with a hypothesis, not a hunch

Before you change anything, write down the belief you’re testing in plain language — and how you’d know if it were wrong.

Frame every test the way you'd frame a question
A weak test

“Let’s try a different subject line and see what happens” gives you little to work with. You haven’t said what you expect, why you expect it or what you’ll do with the result.

A strong test

A clearer hypothesis would be: “We expect subject lines that name a specific benefit to generate more opens than curiosity-led lines among price-sensitive customers, because these buyers look for immediate relevance. We’ll define a worthwhile improvement before testing and assess the result at 95% confidence.”

Why the difference matters

This gives you a clear prediction to test. A meaningful result in either direction can help you refine your approach. If the evidence is inconclusive, you need more data before drawing a conclusion.

Test one variable at a time

Change one variable at a time. If you alter the subject line, send time and main image together, you won’t know which change affected the result. Keep multivariate testing for when you have enough volume to assess several changes reliably.

The toolkit

What's actually worth testing — and in what order

Sample size and statistical caution

The single biggest reason email tests mislead is calling them too early, on too few people. Here’s how to avoid fooling yourself.

Decide the sample size before you send
Before you start, work out how many recipients each version needs to detect a difference worth acting on. If your list is too small, adjust the test or plan to gather results across comparable sends. Set the threshold in advance so you don’t end up treating a small, random difference as a useful finding.
Let the test run its full course
Opens and clicks arrive over hours and days. Calling a winner as soon as one version pulls ahead increases the risk of a false positive. Decide how long the test will run before you launch it, then let it finish.
Treat 95% confidence as the floor, not the finish line
A statistically significant result doesn’t tell you whether the improvement is large enough to matter or likely to last. Look at the confidence interval to understand the range of plausible effects, then consider whether the difference justifies a change.
Beware the multiple-comparisons trap
Running lots of tests also increases the chance of finding a misleading result. At a 5% significance threshold, if there are no real differences, roughly one in twenty tests will appear significant by chance. Choose tests with clear hypotheses and useful consequences for your next decision.
The honest-review rule

A result you can’t replicate isn’t a result. When a test produces a surprising or high-stakes win, re-run it before you rebuild your strategy around it. Our Zero-Blame principle makes this easy: no one is judged for a hypothesis that turned out wrong — only for refusing to check.

Avoiding false conclusions

Even a well-sized, significant test can lead you astray if you misread what it means. These are the traps that quietly erode the value of testing — and how to stay honest about what you actually learned.

The false conclusions that waste your spend

Each of these feels like a finding. None of them is, on its own.

Optimising the wrong metric

  • A subject line wins opens but loses clicks
  • A CTA wins clicks but loses conversions
  • A send wins engagement but drives unsubscribes
Always test against the outcome furthest down the funnel you can reliably measure. Opens and clicks are means, not ends. If a 'winner' lifts a vanity metric while hurting revenue or retention, it lost. Marketing serves the whole business, not its own scoreboard.

Confusing one win with a rule

  • A single Tuesday send outperforms once
  • Emojis win in one campaign
  • Short copy beats long — this time
One result is a data point, not a law. Audience behaviour drifts and context matters. Hold conclusions loosely, re-test the ones you build on, and document what you learned in language a non-marketer could follow — so it becomes shared wisdom, not someone's confident anecdote.

Ignoring who was in the sample

  • Testing only on your most engaged subscribers
  • Mixing new and long-standing contacts
  • Reading one segment's result as everyone's
A result from your keenest openers won't generalise to your whole list. Segment-aware testing — see our guide to list segmentation — tells you what works for whom, which is far more useful than a blurred average that fits nobody.

Declaring a win on a tie

  • A 0.3% gap with no significance
  • Overlapping confidence intervals
  • A difference inside the margin of error
'No detectable difference' is a legitimate and valuable outcome — it tells you to stop spending effort there and focus elsewhere. Forcing a winner out of a statistical tie is how teams convince themselves of things that aren't true.

Email A/B testing: common questions

Straight answers to the questions teams ask most once they start testing properly.
There’s no single number — it depends on the size of the effect you’re trying to detect. As a rough guide, you want at least around 1,000 recipients per variant to detect meaningful differences in open and click rates with any confidence. Below that, reserve testing for big, decisive changes (offer, framing) rather than cosmetic ones, and lean on qualitative judgement for the rest.
Long enough for the behaviour you’re measuring to play out fully — usually a few days for opens and clicks, longer if you’re tracking downstream conversion. Decide the duration before you send and don’t stop early just because one variant pulls ahead. Early stopping is one of the most common causes of false positives in email testing.
Whatever would most change your decisions if you knew the answer. For most senders that’s the offer and its framing, because it moves revenue — not the subject line, which only moves opens. Start with the variable whose result would alter what you do next, and isolate it so the result is interpretable.
It means that, if there were genuinely no difference between your variants, you’d see a result this strong by chance only about 5% of the time. It is reassurance that the effect is probably real — but it says nothing about how large or durable the effect is. Always read the size of the difference alongside the confidence level before acting.
Only if you have the volume for proper multivariate testing, which needs substantially more recipients. For most lists, changing one variable at a time is the honest choice — otherwise a lift can’t be attributed to any single change, and you learn nothing you can reuse.
That’s a genuine result, not a failure. It tells you the variable you tested doesn’t meaningfully move the needle for this audience, so you can stop optimising it and put your effort somewhere with more leverage. Document it and move on — a ‘no difference’ finding saves you future wasted spend.

Where testing fits in your wider email strategy

A/B testing is the discipline that keeps the rest of your email programme honest. It works best alongside the other building blocks of a data-driven approach.

Testing connects to everything else you send

Start with the strategy

Useful tests start with clear questions and a plan for acting on the answers. Our Email Marketing Strategy for Ecommerce guide explains how to build that plan.

Test within segments

Results across your whole list can hide differences between customer groups. A change that helps one group may put another off. Use Email List Segmentation alongside testing to understand who responds to each approach.

 

Test the ethics too

You can also test send times and frequency within your wider campaign schedule. See Email Campaign Calendar for help planning messages that build anticipation, then identify which timing and frequency decisions are worth testing.

Test the rhythm of your calendar

Send timing and frequency are testable, but they live inside a plan. See Email Campaign Calendar for building anticipation across sends — then test the variables that calendar exposes.

Why we test the way we do

Verve stopped us chasing open-rate wins that never showed up in sales. They made us write down a hypothesis for every test and review it honestly afterwards — half of what we 'knew' turned out to be noise, and the decisions we kept were ones we could actually trust.
Head of Ecommerce
Long-standing Verve client, consumer brand

Build an email programme that learns

If your tests aren’t giving you useful answers, we can help. We’ll work with you to define clear hypotheses, choose suitable sample sizes and review the results openly, so each test helps you understand your customers and improve future campaigns.

Hypothesis-led testing

Every test states what it's testing and how you'd know if you're wrong.

Honest, structured review

Think, Plan, Do, Review — so wins become wisdom, not folklore.

Outcomes, not vanity metrics

We measure what moves the business, not what flatters the dashboard.