A/B Testing Emails: How to Run Tests That Actually Teach You Something
A/B testing helps you learn what your audience responds to, so you can make better decisions in future campaigns. That starts with a clear hypothesis: what do you expect to happen, and why?
Without one, even a winning result can be hard to explain. Too small a sample can make chance look like a meaningful difference. And if you only pay attention to results that support your expectations, you miss evidence that could improve your approach. This guide explains how to avoid all three problems.
Start with a hypothesis, not a hunch
Before you change anything, write down the belief you’re testing in plain language — and how you’d know if it were wrong.
“Let’s try a different subject line and see what happens” gives you little to work with. You haven’t said what you expect, why you expect it or what you’ll do with the result.
A clearer hypothesis would be: “We expect subject lines that name a specific benefit to generate more opens than curiosity-led lines among price-sensitive customers, because these buyers look for immediate relevance. We’ll define a worthwhile improvement before testing and assess the result at 95% confidence.”
This gives you a clear prediction to test. A meaningful result in either direction can help you refine your approach. If the evidence is inconclusive, you need more data before drawing a conclusion.
Change one variable at a time. If you alter the subject line, send time and main image together, you won’t know which change affected the result. Keep multivariate testing for when you have enough volume to assess several changes reliably.
What's actually worth testing — and in what order
-
Subject line and preview textThe highest-leverage early test, because it governs whether the email gets opened at all. Test specificity vs. curiosity, length, personalisation, and the preview text that sits beside it. But remember: open rate is a means, not the goal — a line that wins opens but loses clicks has failed.
-
The offer and the askFar more decisive than cosmetics. Test framing (benefit vs. feature), the strength and honesty of any scarcity, and whether a single clear call to action beats several competing ones. Changes here move revenue, not just engagement.
-
Call-to-action copy and placementSmall wording and position changes can shift click-through meaningfully. Test verb choice, first-person vs. second-person, and above-the-fold vs. end-of-email placement. Tie the result to clicks and downstream conversion, not clicks alone.
-
Send time and dayWorth testing once the bigger levers are settled, and best done as a repeatable experiment rather than a one-off — audience rhythms shift. Resist over-reading a single result; send-time wins are among the least stable.
-
Sender name and from-addressA named person vs. a brand can change open and reply behaviour, and it compounds trust over time. Low effort, occasionally high impact — but only test it when you can leave it stable long enough to read the effect.
-
What not to bother testing yetIf your list is small, cosmetic tweaks (button colour, tiny layout shifts) will never reach significance and will burn cycles. Prioritise tests whose outcome would actually change a decision you’re going to make.
Sample size and statistical caution
The single biggest reason email tests mislead is calling them too early, on too few people. Here’s how to avoid fooling yourself.
A result you can’t replicate isn’t a result. When a test produces a surprising or high-stakes win, re-run it before you rebuild your strategy around it. Our Zero-Blame principle makes this easy: no one is judged for a hypothesis that turned out wrong — only for refusing to check.
Even a well-sized, significant test can lead you astray if you misread what it means. These are the traps that quietly erode the value of testing — and how to stay honest about what you actually learned.
The false conclusions that waste your spend
Optimising the wrong metric
- A subject line wins opens but loses clicks
- A CTA wins clicks but loses conversions
- A send wins engagement but drives unsubscribes
Confusing one win with a rule
- A single Tuesday send outperforms once
- Emojis win in one campaign
- Short copy beats long — this time
Ignoring who was in the sample
- Testing only on your most engaged subscribers
- Mixing new and long-standing contacts
- Reading one segment's result as everyone's
Declaring a win on a tie
- A 0.3% gap with no significance
- Overlapping confidence intervals
- A difference inside the margin of error
Email A/B testing: common questions
Where testing fits in your wider email strategy
A/B testing is the discipline that keeps the rest of your email programme honest. It works best alongside the other building blocks of a data-driven approach.
Testing connects to everything else you send
Useful tests start with clear questions and a plan for acting on the answers. Our Email Marketing Strategy for Ecommerce guide explains how to build that plan.
Results across your whole list can hide differences between customer groups. A change that helps one group may put another off. Use Email List Segmentation alongside testing to understand who responds to each approach.
You can also test send times and frequency within your wider campaign schedule. See Email Campaign Calendar for help planning messages that build anticipation, then identify which timing and frequency decisions are worth testing.
Send timing and frequency are testable, but they live inside a plan. See Email Campaign Calendar for building anticipation across sends — then test the variables that calendar exposes.