Email Testing for Beginners – Complete Guide

Wondering where to start with email tests? Not sure how to make your templates highly deliverable? No worries, I got you covered. 

This article guides you through email testing for beginners including definitions, types of testing, tools, and tips for best practices. 

What is email testing?

Email testing is running your email template through testing tools and applying specific methodologies (e.g. A/B testing) to serve two purposes: 

1. Catch errors before they reach your audience (use Mailtrap Email Testing).  
2. Gather insights for improving future email campaigns (use Mailtrap Email Sending or other tools)

The role of email testing explained

1. Error prevention: Popular email clients update frequently, leading to emails rendering incorrectly or not reaching the inbox. Even small errors, like a broken link, can significantly impact your campaign’s ROI. And it’s not uncommon for recipients to unsubscribe from an email just because it didn’t display or work well. Mind you – these errors will also affect your transactional emails, making it all the more important to send bug-free emails. 

2. Gathering insights: Beyond just checking for rendering issues, email testing allows you to conduct tests on elements like subject lines, providing valuable insights into what resonates best with your audience. This information is crucial for crafting more impactful emails. It helps improve email engagement which, in turn, improves your sender’s reputation and overall deliverability.

Types of email testing

Generally, I’d argue that there are three main types of email testing:

1. Sandbox tests 
2. Pre-production tests
3. Production tests 

Sandbox tests happen in an isolated environment, usually via a fake SMTP, and give you an accurate estimation of the following:

• How your email template performs against popular mailbox providers 
• If the template is going to trigger spam filters 
• If your IP or domain appears on a blacklist  

Pre-production tests can, for example, involve DNS authentication tests to ensure your SMTP server is properly authenticated. Or you can run server-side load testing to see how the SMTP handles the traffic. 

Note that I won’t go into the details of pre-production tests for two reasons. 

1. These aren’t for beginners as they require extensive knowledge of email infrastructure.
2. I already covered pre-production testing tools in the linked article ◀️. 

However, I’ll speak about the production tests which may include A/B testing of various email elements, multivariate tests, email heat maps, etc. 

What email components should be tested?

All email components should be tested, including:

• Email template (HTML/CSS)
• Headers 
• Subject 
• Body
• Links
• Spam score 

Now, I want to make the whole story more granular by providing basic testing categories. It’ll help you better understand the testing process and which component tests you should run first. Also, I’ll leave explicit references if something should be done in a sandbox or on production. 

Quality assurance tests

1. Spam testing (sandbox): This step is critical to prevent your emails from landing in spam folders. If email service providers mark your messages as spam, it can hurt the deliverability of your future emails.

2. Design testing (sandbox): This test checks the format and image rendering across different apps and devices.

3. HTML testing (sandbox): This test inspects the HTML and CSS support for your email template and it’s tied to design and content testing. You should clearly see which template element might be problematic and how to fix it, plus have an overall market support score. 

4. Link validating (sandbox): This involves testing every link in your email to ensure they point to the correct locations. It’s essential for the effectiveness of marketing and transactional emails. It helps avoid broken or incorrect links, which can harm your brand’s reputation and subscriber experience. Mailtrap allows you to do it in a sandbox. 

Variable tests

Be careful with multivariate and A/B testing as these are production tests. 

If you fail to tactfully change the variables, or if you change too many variables, the test may yield misleading results. 

The same could happen if you run multivariate and A/B testing at the same time. Simply, it could be hard to understand which variable impacted the email engagement rates the most. 

1. Multivariate testing (production): This type of testing involves testing several variable combinations simultaneously. But you shouldn’t tweak the email body content and copy. For example, you might want to try the following:

• Different email templates with the same messaging 
• Combine that with different send times 
• And mix in different recipient segmentation cohorts (by industry, location, etc.)

The goal is to find the best set of variables that align with your audience (or audiences) and their preferences. 

Pro tip: If you want to test different templates with the same messaging make sure to run a sandbox test on each template to ensure it’s highly deliverable. In case it’s not, this will affect the multivariate test results. 

2. A/B testing (production): This is a widely used strategy where you send one variation of your email to a part of your email list (within the same segmentation cohort by industry, location, etc) and a second variation to another group. Again, be careful how you group the recipients, and don’t change too many elements. 

The reactions (opens, clicks, conversions) of these groups determine the winning email variation. A/B testing is effective in identifying which version of the message increases key performance indicators (KPIs). 

Here are the elements to test:

• Subject line: Test different titles to see which one leads to higher open rates.
• Preheader: This is the text/copy pulled from the initial line of your email body and can be crucial for engagement, especially with mobile users.
• Email Copy: Experiment with different hooks, messaging, and tone.
• Images: For image-heavy emails, test file size and optimization, especially for mobile devices. Generally, the image-to-text ratio should be 20%-to-80% tops. 
• Call-to-Action (CTA): Different CTA button colors or text lines can significantly impact click-through rates.

3. Email heat maps (production): These tests map how recipients engage with your emails. They’ll show the scroll depth, which area recipients focus on, and whether interactive elements attract attention. You can use heat maps in combination with multivariate and A/B testing. Tools that include heatmaps are Mailchimp and ActiveCampaign. 

Email testing tools

I have the most experience with Mailtrap, so I’ll cover the tests you can run using our platform. However, if you’d like a more extensive list of testing tools, I’ve already blogged about it, so feel free to click on the linked article 🔼. 

Mailtrap Email Testing

Mailtrap Email Testing is a sandbox to inspect and debug emails in staging, dev, and QA environments before sending them to your recipients. The testing happens on a fake SMTP so there’s no risk of spamming users. 

With Mailtrap, you can do any of the sandbox tests I mentioned earlier. Here’s a quick list of all the features, and I’ll cover the details later in the article. 

• HTML/CSS check
• Spam score check
• Email preview  
• API for QA automation

You also get:

• Ready-to-use integrations in 20 + languages (Ruby, Python, Node.js, etc.)
• Multiple inboxes for different projects and stages
• User management
• SSO 

Mailtrap Email Sending

Mailtrap Email Sending gives you email infrastructure with high deliverability by design. And you can choose SMTP and API to send. But, since you’re a beginner I’d suggest SMTP. Mailtrap also provides all the DNS authentication records for you, so you won’t need any external tools to run pre-production tests to check the records. 

Okay, but what does email sending have to do with testing?

To begin with, you cannot test deliverability without sending emails to real recipients. And, there are specific production tests you can do, and they’re all about understanding the deliverability metrics (delivery, opens, clicks, spam, etc). 

Also, Mailtrap allows you to set email categories to determine which type of email performs best with your audience. It also helps better manage your email campaigns. Plus, you can use Mailtrap email categories for A/B testing. 

More importantly, you get actionable in-depth analytics that helps quickly troubleshoot unexpected issues. There’s a helicopter-view dashboard with all critical metrics in one place. And you get drill-down reports for each email category and major mailbox providers. 

How to test your first emails?

Note that the following guidelines assume that you already set up Mailtrap Email Testing and Email Sending. If not, click the links to get started with each product. 

This how-to covers tests in order of importance, providing tips on the steps to take. 

Deliverability

Deliverability indicates the likelihood of your emails reaching the recipient inbox, not the spam folder. You can run a deliverability test in two stages – first in a sandbox and then on production to a limited number of recipients. 

• Sandbox (Mailtrap) – Select My Inbox under Email Testing, Inboxes tab. Choose your preferred technology under the Integrations drop-down. 

Make sure to customize the code using your email template and body with associated styles. Run the code, (e.g. you can simply run it in Terminal on MacOS) and your email will appear in the inbox almost instantly. Then, inspect the Spam Analysis and HTML Check tabs. 

• Production – If everything looks okay on the sandbox, choose ~100 recipients (preferably on different mailbox providers), send the email to them, and then inspect Stats to see deliverability data (you’re aiming for 90%+ delivered emails). Also, you may want to set a separate email category for production email tests; ◀️ click the link to see how to do it. 

Tip: If you want to check more email deliverability testing tools, click the link ◀️. 

HTML Checker

Mailtrap’s HTML Check gives you a clear and detailed overview of your email template and whether it’s supported by the most popular mailbox providers. Plus, you can see the support across the Web, Desktop, and Mobile devices. 

Better yet, you get the list of all the analyzed elements and the support, or lack thereof, for each element. And if something needs fixing, you’re only a click away from the specific line of code that could be problematic. 

For example, if you see that your <style> element isn’t supported well, click “Show more” and you’ll get quick troubleshooting tips for that element. To jump to the problematic HTML, just click on the number next to the “Found on lines” prompt. 

Attachments

I suggest sending attachments only with transactional emails. If you try to do it with marketing emails, your deliverability could suffer. 

That out of the way, you can test emails with attachments using Mailtrap Email Testing. Just make sure to change the email code so it can accept attachments. 

Then, run the code using the same steps as described in the Deliverability section, and check the Email Testing inbox. 

If all’s well, the email shouldn’t have a high spam score and the attachment should be properly displayed. If not, HTML Check highlights all the problematic areas for you so you can quickly debug the template. 

Different email clients and devices

As mentioned in the HTML Checker section, Mailtrap Email Testing shows you support across the most popular email clients. 

Better yet, you can pick and choose different clients by clicking on their respective checkboxes. And the same goes for Desktop, Mobile, and Web support. 

The cool thing is that this is all done within the same menu, so you’re, for example, just a few clicks away from testing how well your email does against Apple Mail on mobile. 

Pro Tips: 

• Mailtrap Email Sending also gives detailed deliverability data with drill-down reports on different mailbox providers (Google, Google Workspace, Office 365, etc). So make sure to keep an eye on that metric as well. 

• Check our comprehensive guide on how to test emails with different email clients. 

Templates

With Mailtrap, you can also test your email templates with our API, which allows you to easily switch from staging to production environment once you’re ready to start sending.

All you have to do is enable sandbox and specify the inbox ID to receive template test. Then, you can send it through the API in the production environment.

Dynamic content 

Dynamic content tests represent the tests for emails that automatically change based on your recipients/users’ preferences, behavior, demographics, purchase history, etc. 

These tests give you deliverability and email performance estimations for all email customizations based on a particular target avatar or recipient cohort. As such, they aren’t for beginners and typically require additional setup and integration with CRM tools like Salesforce, Marketo, Adobe Campaign Standard, etc. 

But, if you’d like to know more about dynamic email content, our partner from Hubspot, Irina Nica, wrote a great article about it. Feel free to check it out ⬆️ as well as our dedicated video.

Email testing best practices

Google “email testing best practices” and you’ll end up with a bunch of tips on how to do A/B tests. That’s okay, but stop to think about it for a second…

If your email template has a bunch of technical errors, no A/B test is going to give you accurate results. So my first tip for best practice is – always test your template in a sandbox to ensure it has great support with email clients and a low spam score. 

And yeah, all this is about the numbers and percentages, so here are the figures to aim for with sandbox tests. 

• Market support: Anything above 90% market support across all email clients and devices is a good score. I’d also suggest you consider updating your template if the score is around 85%. 

• Spam score: Mailtrap uses the SpamAssassin filter to estimate the spam score of your email. Any score below 5 is considered good. But if you’re sending marketing emails, you may want to aim for a score of ~3. 

On that bombshell, here are some more tips:

• When running production tests, your deliverability rate needs to be above 90%. A score lower than that indicates a high bounce or spam rate. In case of a high bounce rate, you’ll need to check your email list and purge it from inactive addresses. If the spam score is high, you need to go back and inspect the template and infrastructure to determine the reasons. 

• With A/B tests it pays to focus on email subject and preheader testing since these are the elements that drive up your open rate and, thus your senders’ score. The average open rate for marketing emails across all industries is 21%+, while for transactional emails it must be much higher 85%+. 

• If you struggle with engagement rates (email click rates), no worries, the average for marketing emails across different industries is ~3%. But if you want to boost it, test different CTAs (primary and secondary), or try different link placement. Better yet, focus on one engagement per email (e.g. just one link/CTA) and tweak your copy so that recipients clearly understand the desired action. 

• Finally, this goes back to the error prevention I mentioned at the start – make sure you’re up to date with the latest sender requirements from Google, Yahoo, and other mailbox providers. Changes on their side will affect your deliverability metrics unless you continually update and retest email templates. Check the video below for the latest update requirements.
  

Additional reading: 

• How to Check and Improve Email Sender Reputation [2023] 
• Email Domain Reputation: All You Need to Know [2023] 
• What Is Email Feedback Loop and Why Is It Important? 
• Why Using Dummy Email for Testing Just Doesn’t Work 
• List Unsubscribe Header: Why and How to Use | Mailtrap Blog 

Ready, steady, test!

As you can see, email testing for beginners isn’t tricky at all. Just to recap, you should focus first on the sandbox tests and debug your template if necessary. Then move to production tests with the primary goal of improving engagement metrics (open and click rates).