How to A/B Test Emails: Subject Lines, Send Times, CTAs, Content Length and What to Test First
On this page
Why Most Email A/B Tests Are Worthless
Most email marketers run A/B tests. Most of those tests produce results that mean nothing. The problem is not the concept -- A/B testing is the most reliable way to improve email performance. The problem is execution: testing the wrong things, testing with samples that are too small, declaring winners too early, and changing multiple variables at once.
A well-run A/B test isolates one variable, tests it against a meaningful sample, waits for statistical significance before declaring a winner, and applies the learning to future sends. This sounds simple. In practice, it requires discipline that most marketers skip because they are impatient, because their email platform makes it easy to declare a winner with insufficient data, or because they do not understand the statistics behind testing.
What to Test First
The order in which you test matters. Start with the variables that have the biggest impact on outcomes, then work your way to fine-tuning.
Subject lines should be tested first because they determine whether the email is opened at all. A 5% improvement in open rate has a cascading effect on everything downstream (clicks, conversions, revenue). If nobody opens the email, nothing inside it matters. Test subject line length (short vs. long), personalization (with name vs. without), specificity (vague vs. specific), urgency (time-limited vs. evergreen), question vs. statement, and emoji vs. no emoji.
Sender name and from address should be tested second. "John from Example Co" vs. "Example Co" vs. "John Smith" can produce 10-30% differences in open rate. Many email marketers never test this because they set it once and forget it.
CTA (call-to-action) should be tested third. The CTA determines whether an opened email generates a click. Test button vs. text link, button color, CTA copy ("Get Started" vs. "Start Free Trial" vs. "See Pricing"), number of CTAs (one vs. multiple), and CTA placement (above the fold vs. bottom of email).
Send time and day should be tested fourth. The optimal send time varies by audience, and the only way to find it is to test. Test morning vs. afternoon, weekday vs. weekend, and specific hours within those blocks. Note that send time has a smaller impact than subject line or CTA, so test it after you have optimized the higher-impact variables.
Content length and format should be tested last. Short vs. long email, text-only vs. HTML, single-column vs. multi-column, image-heavy vs. text-heavy. These variables have a smaller per-test impact but compound over time.
Setting Up a Valid Test
A valid A/B test has one variable (everything else is identical between version A and version B), random assignment (each subscriber has an equal chance of receiving version A or version B), sufficient sample size (more on this below), and a predetermined success metric (open rate for subject line tests, click rate for CTA tests, conversion rate for content tests).
The most common mistake is changing multiple variables at once. If you test a different subject line AND a different CTA at the same time, and version B wins, you do not know whether the subject line or the CTA caused the improvement. You have learned nothing actionable. Change one thing at a time.
Sample Size and Statistical Significance
This is where most email A/B tests fail. Your email platform declares version B the "winner" after 2 hours with a 2% open rate difference on 500 recipients. Is that a real difference or random noise? Almost certainly noise.
The minimum sample size for a meaningful test depends on the expected difference between versions. For a subject line test where you expect a 2-3 percentage point difference in open rate (e.g., from 20% to 22-23%), you need approximately 3,000-5,000 recipients per version (6,000-10,000 total). For a CTA test where you expect a 1-2 percentage point difference in click rate, you need 5,000-10,000 per version.
If your list is smaller than these minimums, you have two options. Test over multiple sends (run the same test across 3-4 weekly sends and aggregate the results). Or accept that your tests will be directional rather than statistically significant (they suggest a trend but do not prove it).
Most email platforms (Mailchimp, ActiveCampaign, HubSpot, Klaviyo) have built-in A/B testing features that handle randomization and can calculate statistical significance. Use these features rather than trying to manage tests manually.
Testing Subject Lines
Subject line tests are the highest-value, most reliable email tests because open rate is a large, easily measured number with clear attribution (the subject line is the primary determinant of opens).
Run subject line tests by sending version A to 25% of your list and version B to another 25%, waiting for results (typically 4-24 hours depending on your audience), then sending the winning version to the remaining 50%. Most email platforms automate this workflow.
Effective subject line test pairs are specific vs. vague ("3 ways to reduce your AWS bill by 30%" vs. "Tips for saving on cloud costs"), question vs. statement ("Is your email list costing you money?" vs. "Your email list might be costing you money"), personalized vs. generic ("[Name], your Q3 marketing report is ready" vs. "Your Q3 marketing report is ready"), and short vs. long ("Quick question" vs. "Quick question about your marketing strategy for next quarter").
Keep a testing log: a spreadsheet that records every test you run, both versions, sample sizes, results, statistical significance, and what you learned. Over time, this log reveals patterns specific to your audience (your audience prefers questions over statements, or responds better to numbers than to vague promises) that inform future email creation without needing to test every time.
Testing Send Times
Send time tests require a different methodology than content tests. Instead of splitting one send into A and B, send identical emails to equivalent segments at different times and compare results. For example, send to segment A (a random 50% of your list) at 9am Tuesday and segment B (the other 50%) at 2pm Tuesday.
Run the same test over 3-4 weeks to control for variability. A single send time test can be skewed by external factors (a news event that suppressed opens on one day, a holiday that changed behavior). Multiple tests smooth out these effects.
Maintaining List Quality for Accurate Tests
A/B test results are only reliable if your email list is clean. Invalid addresses, spam traps, and disengaged subscribers distort test metrics by inflating bounce rates (which reduce the effective sample size), suppressing open rates (disengaged subscribers who never open skew the overall rate downward), and creating inconsistent results across tests.
Before running high-stakes tests, export your list (CSV), upload to Email Extractor to deduplicate, and run through a verification service. Test on verified, engaged subscribers to get results you can trust.
