Site icon Mailtrap

What to Do When You Have Sent Test Emails to Your Customers

Diana Lepilkina

Everyone is writing about the importance of proper email testing. And we do promote email testing in a pre-production environment, demonstrating how Mailtrap helps you stay on the safe side. But what can you do if you have already sent the wrong emails to customers? 

You can find many recommendations for dealing with email marketing mistakes but can hardly find information about email development and testing failures. 

Mailtrap was inspired by our own email testing disasters. We will share our stories and will give you a list of recommendations on what to do (and what not to do!) to save your reputation in the eyes of both your customers and management. 

The good news is that there are not that many situations when your email testing mistake can cause serious problems heavily affecting your reputation and revenue. The bad news is that there are situations when sending even test emails to the wrong recipients can lead to a literal disaster, such as a data breach. 

Whatever happened, first of all, you need to put out the fire and then enhance your email testing approach to prevent future failures. 

So, how to put out the fire?

STEP 1. Immediately after discovering that your test emails are flying straight to real inboxes, figure out whether you can stop the sending process. 

Large volumes of messages are usually sent in batches to distribute the server loads. It might still be possible to clean up the sending queue. If you are using an email sending provider, contact them ASAP to ask if it’s possible to terminate sending. 

Here is the story behind Mailtrap. Our team was working on the project for a big client, developing a technology review platform. It was also featuring email sending capabilities. The platform had already been released and the user database was quite significant. We had a solid and fine-tuned testing process, set up in a staging environment. That was almost ten years ago, so the variety of the email testing tools was pretty limited. 

Our product manager was experimenting with the implementation of another cool feature (no one remembers what that was exactly) and was sure that everything ran as usual. He had done that a hundred times! Soon, our development team started receiving alerts about non-typical activity on the staging server. Do you know what that was? 200,000 test emails sent out to real users! And that was just the first batch out of five! Most likely, we just uploaded a wrong .csv database with real data.

Luckily, we were using a monitoring system and were able to find and stop this sending process. And fortunately, it was a super simple notification without any personal data. We immediately informed our client about what had happened and sent out an email to affected users with apologies and a request to ignore the previous message. 

No one’s reputation was damaged and everything was fine. But we had learned a great lesson: the staging environment must be connected to a fake SMTP server that collects all emails in virtual inboxes. That’s how Mailtrap was born. 

Try Mailtrap for Free

STEP 2. Find out what had happened, who was affected, and what kind of damage was caused. Check the server logs and/or email sending system reports. You need to do this quickly to be able to respond appropriately. 

STEP 3. Inform your team. The main idea here is to make all related team members aware of the situation to help fix it. The details depend on what actually happened:

Everyone makes mistakes. In most cases, email sending failures happen when you work with well-established processes and stop treating email testing seriously. For example, we have received test emails from such big names as Slack and Braintree. Also, there have been a few email testing disasters, which became public.

STEP 4. Send apologies to your customers. Here the same rule applies: the sooner you notify your customers, the better. The best practice is to use the same subject for the apology email as in the original message (your wrong test email). In this case, your messages are merged into one thread, and in some cases, your customers won’t even read your email failure. 

If you sent a wrong general notification and haven’t revealed any account specific data, then your apology email should be simple (and there is no reason to panic!).

“Hello, please ignore our previous email about the need to update data in your account. It was sent by mistake, and you don’t need to take any action.

Sorry for any inconvenience caused.” 

In complex cases, your apology email should be more personalized. Show empathy and tell what you’re going to do with an emerged issue. Don’t forget: even in severe situations, keep your message simple and concise.

One of the worst cases is causing a data breach with your test emails. If you have unintentionally disclosed personal data, you have to admit it and report the data breach according to the security standards that apply to your business (ex: GDPR, CCPA, HIPAA, etc.). 

What you shouldn’t do in any case when you have sent out test emails to customers

  1. Don’t panic. Even if you released private data like emails and names in your message, it’s already done. Your task is to minimize the damage by acting quickly and in cooperation with your team. Panicking will lead to more mistakes. 
  2. Don’t try to hide the situation. You and your company should act responsibly. Everyone can make a mistake but taking proper responsibility for it adds trust to your brand. Except for a quick reaction from your side, you should provide users with the opportunity to reach you freely. For example, by using live chat functionality, customers can inform you about some issues with wrong emails or so.
  3. Don’t leave everything as it is. You have to take actions to prevent similar mistakes in the future. Making mistakes is fine but learning nothing from it is a complete disaster. 

Email testing failures are not something special. They happen here and there but quite rarely become public (well, except threads on Twitter). It’s time to breath out and address how to keep your emails safe. 

How to run email testing and sending in the right way 

Once you have taken the proper actions to minimize the effect of an email testing failure, you need to work on preventing such situations in future. This is STEP 5 and an extremely important one. 

If you have sent a message to the wrong person once, it means that you are tired. But if you have sent the wrong emails to a user database, it means that your email infrastructure has weak spots. Whether you are a manager, developer, or QA specialist, you must find and improve them. 

The following part of this article will help you cope with this task. 

Email testing process

Inspect every step in what and how you test. To reduce the risk of email testing failures, you have to minimize the human factor. Everything should be automated, and a very limited number of people should have access to your servers and deployment processes. 

Let’s say you have a web service for booking trips. This is a complex system, which has many integrations, dependencies, and a solid user database. 

Of course, you are sending both marketing and transactional emails. To stay competitive, you regularly improve your system’s design and functionality. Make sure you test every kind of change that can affect emailing. Follow these rules: 

What you have to test

The short answer is everything. For transactional messages:

1. Make sure that definite actions trigger the right events: after clicking “confirm my request,” your users will get the appropriate email notifications, for example. 

2. Perform load testing: if 3,500 users will press “confirm my request,” how soon will they be able to receive their email notifications? Are you sure that all of them will be sent out?
Send a batch of emails from each of your servers to different email addresses and then compare the sent and received results. For example, in Mailtrap, it will look as follows:

Each of these Mailtrap inboxes has its own email address, so you can easily monitor the performance in one place. We have explained how to load test your email system in this article. 

3. Make sure that the merging mechanisms work right. Are you sure that the personal data of user A doesn’t go to user’s C email address? Do personalization and localization work as designed?

One of the worst cases is when something is broken inside and you don’t even know about it. You can send wrong invoices or payment notifications to improper users revealing personal financial data, until someone attentive to detail notices and informs you. What if you manage an app with yearly billing and your recipients review invoices after a while? Can you imagine what can happen due to a simple mistake?

Don’t test three random emails out of 1,000. Check them all. You must be sure that every message is right. It’s not possible to manage such a workflow manually, instead you need to use testing automation. With Mailtrap API, you can run Selenium tests and verify the content of every email, notwithstanding the number of messages. 

Sending mass emails is easier to implement and test, as you decide and control when emails are sent. 

1. Make sure that the right email database is selected. There are so many cases when test emails went to the list of real users due to the improper database being loaded. 

2. This sounds obvious but check Bcc. Bcc is still frequently used for sending mass emails in small batches, and once in a while it is confused with Cc.

Several years ago, we received a news digest from one of the local media. It was not personal, just a list of their top performing articles. But the list of recipients was revealed in Cc! This way, we could see emails of other users. In the end, one of the recipients created a database of these media subscribers, and shared it online. This is a data breach!

There is only one character different between Cc and Bcc and can be easily confused in a script. Always check headers and use tools that support Bcc testing. For example, in Mailtrap, you will instantly see whether any email addresses were included in the Bcc field.

In a big team, check who has access to users’ data and the email sending process. 

Start Testing

Deployment process

Both in staging and production, you should limit access and automate processes. 

Email system deployment has to be treated as seriously as any other. 

Final words

If you are reading this article, then you most probably have already dealt with sending wrong emails. 

Long story short, if this happens, first you have to do the following: 

Once done, analyze and enhance your email testing process as such:

Exit mobile version