How to Test Emails in Your Python App

Ketevan Bostoganashvili

11 months ago

Python provides multiple ways of testing emails. It has native options and the ability to integrate with a third-party tool, such as Mailtrap Email Testing.

I’ve recently explored a few approaches and will share my experience in this tutorial. I’ll cover native methods such as unit tests + aiosmtpd, Python’s subprocess function, as well as unit tests’ mock object library.

I’ll also demonstrate how to integrate and test emails in Python with Mailtrap Email Testing.

Native ways of testing emails in Python

Let’s start with the native ways of testing Python emails.

Using unit tests and aiosmtpd

Aiosmtpd is a library that lets you set up a local SMTP (Simple Mail Transfer Protocol) server. This will create a testing environment and handle email traffic internally. So, your test emails won’t reach the recipients. Refer to aiosmtpd’s GitHub page for more information.

I’ll create a test for a simple text email.

Prerequisites:

Python 3.7 and up.

Note: I added the code snippets below in the tests.py file.

To install aiosmtpd, run the pip install aiosmtpd command.

Run the server with python -m aiosmtpd -n command.

I’ll use the unit test framework, a standard library in all versions of Python since 2.1, and the smtplib library. smtplib module uses standard RFC 821 protocol for formatting messages.

from unittest import TestCase
import smtplib

Create a class that will inherit the unit test from the TestCase object and configure the test case.

class EmailTestCase(TestCase):
    def test_send_email(self):
        sender_email = "your_email@gmail.com"
        receiver_email = "receiver_email@example.com"
        message = "This is a test email."

The sender_email and receiver_email variables can be populated with sample data as we simulate email sending. The message variable should contain the desired email body.

The next step is to send a test email using the smtplib.

with smtplib.SMTP(host="localhost", port=8025) as server:
            server.ehlo()
            server.sendmail(sender_email, receiver_email, message)

Here, we create a new instance of an SMTP client. By default, the server works on localhost with port number 8025. We also initiated an SMTP handshake with EHLO and added the sendmail method to send the emails. The sendmail method takes three arguments – sender_email, receiver_email, and message.

The script is now ready. Here’s the full sample:

from unittest import TestCase
import smtplib

class EmailTestCase(TestCase):
    def test_send_email(self):
        sender_email = "your_email@gmail.com"
        receiver_email = "receiver_email@example.com"
        message = "This is a test email."
       
        with smtplib.SMTP(host="localhost", port=8025) as server:
            server.ehlo()
            server.sendmail(sender_email, receiver_email, message)

To run the Python code, use the Run button.

If the test passes, you’ll see a message similar to the one below.

You’ll also see that the email was sent in the terminal.

Limitations of this approach

While this approach can be useful for testing simple email-sending scripts, I found that it has multiple limitations:

Each time I ran the tests, I had to type aiosmtpd manually. This is okay for small and occasional tests, but it’s a huge pain during scaled testing.
As we’re simulating the process of sending emails, we can’t test real-world email delivery.
This setup doesn’t allow for testing high email load.
It doesn’t allow testing advanced email functionalities or features, such as client rendering, for example.

Using Python’s subprocess function with unit tests and aiosmtpd

One way to automate processes while using unit tests and aisosmtpd is Python’s subprocess function. It allows us to run an external program from the Python script.

I first had to import the subprocess function and time module to improve and enhance the previous script. I’ll use the latter to wait for a few seconds before the server is ready.

from unittest import TestCase
import smtplib
import subprocess
import time

Then, I added the setUp method. It prepares the environment to run the server. subprocess.Popen function will execute the command in a new process.

In this case, the command is exec python -m aiosmtpd -n, meaning that the server will run in a new process.

shell-True will allow us to execute the command using the shell. We won’t have to create a new terminal. Rather, the process will run in the background.

As mentioned, time.sleep(2) will pause the execution of the script for 2 seconds to give the server enough time to be ready.

def setUp(self):
    self.process = subprocess.Popen(args='exec python -m aiosmtpd -n', shell=True)
    time.sleep(2)

The next step is to add a tearDown method, which will terminate the subprocess and wait for the process to finish termination.

def tearDown(self):
    self.process.kill()
    self.process.wait()

The test itself is essentially the same, but I added one more assertion. It checks if the server is working and the socket is open.

self.assertIsNotNone(server.sock)

Here’s the complete code sample:

from unittest import TestCase
import smtplib
import subprocess
import time

class EmailTestCase(TestCase):
    def setUp(self):
        self.process = subprocess.Popen(args='exec python -m aiosmtpd -n', shell=True)
        time.sleep(2)

    def test_send_email(self):
        sender_email = "your_email@gmail.com"
        receiver_email = "receiver_email@example.com"
        message = "This is a test email."
        with smtplib.SMTP(host="localhost", port=8025) as server:
            server.ehlo()
            server.sendmail(sender_email, receiver_email, message)
            self.assertIsNotNone(server.sock)

    def tearDown(self):
        self.process.kill()
        self.process.wait()

At this point, we can run the script with the Run button.

The tests were successful, meaning the server was turned on and off as expected.

Running the script from the console

As you’ll notice, I used the Run button to run the scripts in the previous examples. I’ll add the if statement to run the code from the console, modify the unit test import, and reference the unittest.TestCase with the EmailTestCase class definition.

import unittest
import smtplib
import subprocess
import time

class EmailTestCase(unittest.TestCase):

    def setUp(self):
        self.process = subprocess.Popen(args='exec python -m aiosmtpd -n', shell=True)
        time.sleep(2)

    def test_send_email(self):
        sender_email = "your_email@gmail.com"
        receiver_email = "receiver_email@example.com"
        message = "This is a test email."

        with smtplib.SMTP(host="localhost", port=8025) as server:
            server.ehlo()
            server.sendmail(sender_email, receiver_email, message)
            self.assertIsNotNone(server.sock)

    def tearDown(self):
        self.process.kill()
        self.process.wait()

if __name__ == '__main__':
    unittest.main()

With the updated setup, we can now run the code directly from the console using the python tests.py command.

Testing if the script can read files

The last modification in the script was to enable it to read emails from the file. This would allow me to test if the variables in the template were substituted correctly. I went with the simple setup once again.

So, I created a new template file with only a message variable.

I went back to the tests.py file and added a with statement which would be responsible for opening and closing files.

with open('template.html') as file:
    template = file.read()

Using the format function, I added the message to the template.

template = template.format(message=message)

And inserted the template into the sendmail function.

server.sendmail(sender_email, receiver_email, template)

The whole script will look something like this:

import unittest
import smtplib
import subprocess
import time

class EmailTestCase(unittest.TestCase):

    def setUp(self):
        self.process = subprocess.Popen(args='exec python -m aiosmtpd -n', shell=True)
        time.sleep(2)

    def test_send_email(self):
        sender_email = "your_email@gmail.com"
        receiver_email = "receiver_email@example.com"
        message = "This is a test email."

        with open('template.html') as file:
            template = file.read()

        template = template.format(message=message)
        with smtplib.SMTP(host="localhost", port=8025) as server:
            server.ehlo()
            server.sendmail(sender_email, receiver_email, template)
            self.assertIsNotNone(server.sock)

    def tearDown(self):
        self.process.kill()
        self.process.wait()

if __name__ == '__main__':
    unittest.main()

Run the code with python test.py and check the output. The message variable was replaced correctly, so the test was successful.

Limitations of this approach

Similar to using aiosmtpd and unit tests, this expanded approach also has its limitations:

Doesn’t allow for testing how complex HTML will render on mobile or desktop devices;
Doesn’t allow for deliverability testing;
Doesn’t allow for checking client support for HTML emails;
Doesn’t allow for testing the communication between the script and external mail servers;
Relies on aiosmtpd and port 8025. Tests may fail if the server isn’t set up correctly.

Using the unit test’s mock object library

Another option for testing emails in Python natively is the unit test’s mock object library. It lets you mock the SMTP server connection without sending the emails.

Here’s the script:

import unittest
from email.mime.text import MIMEText
from unittest.mock import patch
import smtplib

def send_email(server, port, subject, message, from_addr, to_addr):
   
    smtp_user = 'username'
    smtp_password = 'password'
    msg = MIMEText(message)
    msg['From'] = from_addr
    msg['To'] = to_addr
    msg['Subject'] = subject
    with smtplib.SMTP(server, port) as server:
        server.starttls()
        server.login(smtp_user, smtp_password)
        server.send_message(msg)
class TestEmailSending(unittest.TestCase):
    @patch('smtplib.SMTP')
    def test_send_email(self, mock_smtp):
        # Arrange: Setup our expectations
        subject = "Test Subject"
        message = "Hello, this is a test."
        from_addr = 'from@example.com'
        to_addr = 'to@example.com'
        server = "sandbox.smtp.mailtrap.io"
        port = 587
        # Act: Call the send_email function
        send_email(server, port, subject, message, from_addr, to_addr)
       
        # Assert: Check if the right calls were made on the SMTP object
        mock_smtp.assert_called_with(server, port)
        instance = mock_smtp.return_value.__enter__.return_value
        instance.send_message.assert_called_once()
        call_args = instance.send_message.call_args[0]
        sent_email = call_args[0]
       
        # Verify the email content
        self.assertEqual(sent_email['Subject'], subject)
        self.assertEqual(sent_email['From'], from_addr)
        self.assertEqual(sent_email['To'], to_addr)
        self.assertEqual(sent_email.get_payload(), message)

if __name__ == '__main__':
    unittest.main()

The provided code defines a function and includes a test class, TestEmailSending, using Python’s unit test framework to test this function.

The send_email function takes server details, subject, message body, sender’s address, and recipient’s address as parameters, creates an email (MIMEText) object with these details, and then logs into an SMTP server to send it.

In the test case, the smtplib.SMTP class is mocked using the unittest.mock.patch, allowing the test to verify that the SMTP server is called with the correct parameters without actually sending an email.

The test checks if the send_message method of the SMTP instance is called correctly and asserts that the email’s subject, from address, to address, and payload match the expected values.

Limitations of this approach

The mock object library isn’t sufficient for detecting issues with implementation;
Doesn’t allow for checking client support for HTML emails;
Doesn’t allow for testing how complex HTML will render on mobile or desktop devices.

How to test emails in Python with Mailtrap via SMTP

To address the limitations of the native methods, I decided to use Mailtrap Email Testing. It’s an Email Sandbox that provides a safe environment for inspecting and debugging emails in staging, QA, and dev environments.

I used Email Testing to run functional and deliverability tests, check email formatting, analyze the content for spam, check attachments, and test error handling.

You’ll need an account to use it, and you can quickly create it by opening the signup page. Once the account is up and running, we can go straight to testing.

Functional testing

We’ll need a slightly different script in this case. Create a new EmailTestCase class and add sample data. Here, test_mailtrap is a method that will send a test email to Mailtrap.

import unittest

class EmailTestCase(unittest.TestCase):

    def test_mailtrap(self):
        sender_email = 'sender@example.com'
        recipient_email = 'recipient@example.com'
        subject = 'Test Email'
        body = 'This is a test email sent from the SMTP server.'

The next step is to import the built-in EmailMessage object.

from email.message import EmailMessage

Using it, we’ll build a message and add the sample data to it.

msg = EmailMessage()
msg['From'] = sender_email
msg['To'] = recipient_email
msg['Subject'] = subject
msg.set_content(body)

The last step is to connect to Mailtrap Email Testing’s fake SMTP server. For that, I’ll use smtplib and the with statement. At this point, I’ll need Email Testing’s SMTP credentials. To access them, go to Email Testing → My Inbox → SMTP Settings and click Show Credentials.

Copy the Host, Username, and Password, and return to the Python project. Create a new instance of the SMTP client and connect to the server using the host and port.

Call the server.login method and pass a username and password to it.

with smtplib.SMTP(host="sandbox.smtp.mailtrap.io", port=2525) as server:
    server.login(user="your_username", PASSWORD)
    server.send_message(msg)

Note: Your_username is a placeholder. You should add your actual username here.

For security reasons, I’ve stored the password in a separate folder. So, I’ll import it.

from variables import PASSWORD

The email-testing script is now ready.

import unittest
import smtplib

from email.message import EmailMessage

from variables import PASSWORD

class EmailTestCase(unittest.TestCase):

    def test_mailtrap(self):
        sender_email = 'sender@example.com'
        recipient_email = 'recipient@example.com'
        subject = 'Test Email'
        body = 'This is a test email sent from the SMTP server.'
        msg = EmailMessage()
        msg['From'] = sender_email
        msg['To'] = recipient_email
        msg['Subject'] = subject
        msg.set_content(body)

        with smtplib.SMTP(host="sandbox.smtp.mailtrap.io", port=2525) as server:
            server.login(user="your_username", PASSWORD)
            server.send_message(msg)

if __name__ == '__main__':
    unittest.main()

I ran the code with the python tests.py command. And the email arrived in the Email Testing inbox within seconds.

Test results:

Verified whether the EmailMessage object was correctly populated with the sender’s email, recipient’s email address, subject, and body text. I checked the body by going to the Text tab and verified from, to, and subject from the Tech Info and Raw tabs.

Revealed that the code established an SMTP connection successfully.
Revealed that the SMTP authentication process was correct.
Confirmed that the text message formatting was correct and, as a result, it was successfully sent through the SMTP server.
Verified that send_message method of the smtplib.SMTP class was called with the correct parameters.

Checking email formatting and HTML code

Next up in my tests were:

Checking whether the message body in plain text and HTML versions would be the same;
Checking the mobile, desktop, and tablet previews;
Analyzing HTML code to see if it contained any unsupported elements.

For that, I sent a simple HTML email to my virtual inbox using the following lines of code. I used an email-sending script, not a unit test in this case, since the email would still get captured in the inbox. For more information on sending HTML emails from Python, refer to this blog post.

import smtplib
from email.mime.text import MIMEText
from email.mime.multipart import MIMEMultipart

port = 2525
smtp_server = "sandbox.smtp.mailtrap.io"
login = "your_username"
password = "your_password"

sender_email = "sender@example.com"
receiver_email = "recipient@example.com"
message = MIMEMultipart("alternative")
message["Subject"] = "HTML test"
message["From"] = sender_email
message["To"] = receiver_email

# write the text/plain part
text = """\
Hi,
Check out the new post on the Mailtrap blog:
SMTP Server for Testing: Cloud-based or Local?
https://blog.mailtrap.io/2018/09/27/cloud-or-local-smtp-server/
Feel free to let us know what content would be useful for you!"""

# write the HTML part
html = """\
<html>
  <body>
    <p>Hi,<br>
      Check out the new post on the Mailtrap blog:</p>
    <p><a href="https://blog.mailtrap.io/2018/09/27/cloud-or-local-smtp-server">SMTP Server for Testing: Cloud-based or Local?</a></p>
    <p> Feel free to <strong>let us</strong> know what content would be useful for you!</p>
  </body>
</html>
"""

# convert both parts to MIMEText objects and add them to the MIMEMultipart message
part1 = MIMEText(text, "plain")
part2 = MIMEText(html, "html")
message.attach(part1)
message.attach(part2)

with smtplib.SMTP("sandbox.smtp.mailtrap.io", 2525) as server:
    server.login(login, password)
    server.sendmail(
        sender_email, receiver_email, message.as_string()
    )

print('Sent')

Test results:

Verified that the HTML message wasn’t broken on mobile, desktop, or tablet views.

Confirmed that the text content was the same as the HTML content.

Found out that my email contained one element that email clients would support partially or wouldn’t support at all.

Deliverability testing

To extend my tests, I opened the Spam Analysis tab. This showed the Spam Report with spam score, as well as spam points with descriptions. This is useful for improving the template if it exceeds the threshold of 5. Spam tests are run with the help of the SpamAssassin filter.

Test results:

Verified that the spam score of the email was below the threshold of 5.

Checked the Blacklist Report to see if my domain would be listed in any of the blocklists.

Verified that my emails would be delivered to the recipients’ inboxes if I were to send them on prod.

Checking the attachments

I also wanted to see if the email-sending script would attach, encode, and send attachments correctly. Attachments are MIME objects. Encoding implies using a base64 module that encodes all binary data with ASCII characters.

I simply added Email Testing SMTP credentials to this script and waited for the email to arrive in the virtual inbox.

import smtplib

from email import encoders
from email.mime.base import MIMEBase
from email.mime.multipart import MIMEMultipart
from email.mime.text import MIMEText

port = 2525
smtp_server = "sandbox.smtp.mailtrap.io"
login = "your_username" 
password = "your_password"

subject = "An example of boarding pass"
sender_email = "mailtrap@example.com"
receiver_email = "new@example.com"

message = MIMEMultipart()
message["From"] = sender_email
message["To"] = receiver_email
message["Subject"] = subject

body = "This is an example of how you can send a boarding pass in attachment with Python"
message.attach(MIMEText(body, "plain"))

filename = "yourBP.pdf"

with open(filename, "rb") as attachment:
    part = MIMEBase("application", "octet-stream")
    part.set_payload(attachment.read())

encoders.encode_base64(part)

part.add_header(
    "Content-Disposition",
    f"attachment; filename= {filename}",
)

message.attach(part)
text = message.as_string()

with smtplib.SMTP("sandbox.smtp.mailtrap.io", 2525) as server:
    server.login(login, password)
    server.sendmail(
        sender_email, receiver_email, text
    )
print('Sent')

This code snippet sends an email with a PDF attachment. It assumes that the file is located in the same directory in which you run your Python script. The attach method adds the attachment to the message and converts it into a string.

With a similar method, you could also test other types of attachments. In that case, you should simply use an appropriate class such as email.mime.audio.MIMEAudio or email.mime.image.MIMEImage. Read Python docs for more information.

This is what it will look like in the Email Testing inbox.

Test results:

Verified that the script sends an email with an attachment successfully;
Verified that the file path was configured properly.

Testing templates

Next, I tested my email templates with Mailtrap API, which lets me switch from staging to production once I’m ready to start sending.

All I did was enable sandbox, specify inbox ID to receive the template test, and then send it through the API in the production environment.

Testing error handling

The final test I ran with SMTP was to check error handling. I sent emails using the smtplib module once again, but this time with try and except blocks.

import smtplib
from socket import gaierror

port = 2525
smtp_server = "sandbox.smtp.mailtrap.io"
login = "your_username"
password = "your_password"
sender = "sender@example.com"
receiver = "recipient@example.com"
message = f"""\
Subject: Test email
To: {receiver}
From: {sender}

This is my test message with Python."""

try:
    with smtplib.SMTP(smtp_server, port) as server:
        server.login(login, password)
        server.sendmail(sender, receiver, message)
    print('Sent')
except (gaierror, ConnectionRefusedError):
    print('Failed to connect to the server. Bad connection settings?')
except smtplib.SMTPServerDisconnected:
    print('Failed to connect to the server. Wrong user/password?')
except smtplib.SMTPException as e:
    print('SMTP error occurred: ' + str(e))

This script will:

Catch gaierror and ConnectionRefusedError exceptions if there are issues connecting to the SMTP server;
Catch smtplib.SMTPServerDisconnected exception if the server disconnects unexpectedly (when login credentials are invalid, for example);
Catch smtplib.SMTPException exception for all the other SMTP errors.
Print the specific error message received from the server.

Start Testing Python Emails with Mailtrap

How to test emails in Python with Mailtrap via API

Mailtrap Email Testing also provides an option to test your emails using API. It’s based on REST principles and returns calls as JSON objects. All the details about the Email Testing API are covered in the API docs.

To make API requests, you’ll need your API token and inbox ID.

Go to Settings → API Tokens, and click Add Token.

Create a token for the desired project and inbox.

Once the token is ready, go to the desired inbox. Check the URL – the inbox ID is the 7-digit number between inboxes/ and /messages.

A sample API request looks something like this:

import http.client

conn = http.client.HTTPSConnection("sandbox.api.mailtrap.io")

payload = "{\n  \"to\": [\n    {\n      \"email\": \"john_doe@example.com\",\n      \"name\": \"John Doe\"\n    }\n  ],\n  \"cc\": [\n    {\n      \"email\": \"jane_doe@example.com\",\n      \"name\": \"Jane Doe\"\n    }\n  ],\n  \"bcc\": [\n    {\n      \"email\": \"james_doe@example.com\",\n      \"name\": \"Jim Doe\"\n    }\n  ],\n  \"from\": {\n    \"email\": \"sales@example.com\",\n    \"name\": \"Example Sales Team\"\n  },\n  \"attachments\": [\n    {\n      \"content\": \"PCFET0NUWVBFIGh0bWw+CjxodG1sIGxhbmc9ImVuIj4KCiAgICA8aGVhZD4KICAgICAgICA8bWV0YSBjaGFyc2V0PSJVVEYtOCI+CiAgICAgICAgPG1ldGEgaHR0cC1lcXVpdj0iWC1VQS1Db21wYXRpYmxlIiBjb250ZW50PSJJRT1lZGdlIj4KICAgICAgICA8bWV0YSBuYW1lPSJ2aWV3cG9ydCIgY29udGVudD0id2lkdGg9ZGV2aWNlLXdpZHRoLCBpbml0aWFsLXNjYWxlPTEuMCI+CiAgICAgICAgPHRpdGxlPkRvY3VtZW50PC90aXRsZT4KICAgIDwvaGVhZD4KCiAgICA8Ym9keT4KCiAgICA8L2JvZHk+Cgo8L2h0bWw+Cg==\",\n      \"filename\": \"index.html\",\n      \"type\": \"text/html\",\n      \"disposition\": \"attachment\"\n    }\n  ],\n  \"custom_variables\": {\n    \"user_id\": \"45982\",\n    \"batch_id\": \"PSJ-12\"\n  },\n  \"headers\": {\n    \"X-Message-Source\": \"dev.mydomain.com\"\n  },\n  \"subject\": \"Your Example Order Confirmation\",\n  \"text\": \"Congratulations on your order no. 1234\",\n  \"category\": \"API Test\"\n}"

headers = {
    'Content-Type': "application/json",
    'Accept': "application/json",
    'Api-Token': "your_api_token"
}

conn.request("POST", "/api/send/your_inbox_id", payload, headers)

res = conn.getresponse()
data = res.read()

print(data.decode("utf-8"))

Once again, examine the API docs and this page, in particular, to learn how to send API requests in Python (or other languages), send a sample request, and check out the sample response.

Wrapping up

Python’s native methods are sufficient to test email-sending and run other functional tests. However, to check the spam score, the connection with external servers, or HTML/CSS, you’ll need a dedicated testing tool such as Mailtrap Email Testing.

If you want to learn more about email-sending and testing in Python, then check out our blog posts:

We’ve also covered sending emails in popular Python frameworks, such as Django and Flask.

Keep an eye on our blog as we publish more testing topics, such as how to test emails, email testing for beginners, email sandboxing, and others.

Don’t let Python’s snake crush your emails. Good luck!