SMTP Security: Best Practices and Top Issues

What do you send in your emails? Pictures from your last trip or recipes for your kids? Business reports for your supervisor? Or maybe something even more confidential like login details or social security numbers? We’re not here to judge. But whatever you include in your emails, you definitely don’t want anyone but the receiver to see. Luckily for all of us, nearly all email clients take care of encrypting our emails so you don’t have to worry about things like secure SMTP or SSL/TLS. But if you want to make a more mindful decision or you’re just setting up your own client, it’s good to know what really happens under the hood. Let’s see!

Ready to deliver your emails?

Try Mailtrap for Free

What’s SMTP? Is it secure?

Simple Mail Transfer Protocol is the technology used by the vast majority of email clients to move messages between servers, on the way to the end-users. In simple words – it’s what gets your emails to the recipient within seconds, even if they’re currently chilling in the 142 bus at Stampede Trail in Alaska (LTE coverage might be a factor, though).

Think of SMTP as a digital postman that first, picks up your package (email) and takes it from your email client to the server, aptly named SMTP server (as it’s used solely for sending out messages). Another postman then picks up the package and takes it all the way to the recipient’s server where it’s picked up and stored for milliseconds at the recipient’s POP3/IMAP server. When all the pleasantries are exchanged, the incoming mail is then delivered to the recipient, without the involvement of SMTP anymore but with separate IMAP/POP3 protocols.

As you can see, SMTP takes care of a significant portion of each outgoing transmission. Given the importance and omnipresence of this protocol, you would think it’s heavily encrypted and secured with top-notch technologies from the secret Google or Yahoo! labs. Well, none of that. The standard SMTP protocol comes with no security features, making it really vulnerable to hijacking and other forms of attacks. It’s like our postman boarded a public intercity bus, dropped a bag with mail on one of the seats and got off right away. Will it arrive intact to a receiver? Possibly. Will it be easy to check what’s being sent or, well, play with it a bit to your or recipient’s disadvantage? Likely too.

What are the threats to SMTP security?

There’s a number of things to be aware of when transmitting even a small amount of emails. Here are the most commonly occurring ones:

Unauthorized access to your emails and data leakage

Cybercriminals might try to get access to your SMTP server that all the outgoing mail goes through. This is done by breaking your authentication procedures or less sophisticated methods, such as accessing the server through hacking employee devices that aren’t included in a network discovery protocol or have accessed the mail server from an unsecured network. When in, unwanted visitors can access your emails and use them to their advantage, by for example leaking your users’ data or stealing confidential information you were sending to coworkers.

This is why it’s crucial to implement things such as SMTP authentication which we cover in detail in this Mailtrap tutorial:

Spam and Phishing

When fraudsters are able to access your SMTP server, they’re also likely to use it to send unauthorized messages to both your contacts and external accounts (this is known as using your server as an Open Relay). This is done to send spam which, when sent from your legitimate and (likely) well-known domain, might be quite successful. Or, even worse, your server can be used to send malicious emails, for example requesting your users share their login credentials or credit card numbers.

Malware

Attackers commonly use the vulnerabilities of SMTP to spread malicious software to the recipients of your email but also in your own infrastructure. These can be viruses, Trojan horses or any other types of worms that are then used to obstruct operations, gain access to servers, change privileges and access secure data. If not fought with sufficient force, malware might continue spreading, infecting more and more servers and users.

DoS attacks

If all of the above didn’t seem serious, cybercriminals can also use your SMTP server to perform a Denial-of-Service (DoS) attacks. This basically means flooding other servers with a huge amount of emails to affect their performance or even cause a crash. DoS can also be used to flood an inbox to hide any warning messages about security breaches to a server. Whatever a DoS attack’s purpose, it’s never good.

These are only some of the examples. But is there a way to secure our connection and avoid such a fate?

How to make SMTP secure? What’s SSL/TLS?

Email providers also noticed these vulnerabilities and from the early days of email started adding security layers to SMTP protocol. SSL (Secure Sockets Layer) was developed back in 1995 by Netscape. The first release (1.0) was never shared with the public due to its vulnerabilities but v2.0 quickly became the must-have feature for every respectable email client.

A few years later, a new standard called TLS (Transport Layer Security) was released and has been constantly improved ever since. SSL was finally deprecated in 2015 and TLS in its 1.3 release (the latest one at the time of writing) is now considered an industry standard. Although it’s still possible to use the latest versions of SSL, it’s not recommended due to various vulnerabilities that have been discovered. Note, however, that names ‘SSL’ and ‘TLS’ are used interchangeably and often a service provider might be referring to SSL while, in fact, they use TLS.

SSL/TLS provides a way to encrypt messages exchanged between your email client and email server. When hackers breach the security of SMTP, they’ll only see a set of seemingly random characters replacing the content of emails. They can still use their newly gained powers to cause damage but at least you and your contacts’ data will be protected.

TLS also supports the use of digital certificates that provide an extra layer of security. Think of these certificates as IDs or passports that each side of the process identifies themselves with. This process is called a Handshake.

How does the Handshake work?

We established already that any email sent over SMTP travels the following way:

sender’s email client
sender’s email server
recipient’s email server
recipient’s email client

When an email crosses between either of these stages, a connection is kicked off and, when using TLS, both sides need to establish trust between each other. This is called the Handshake. The sender’s client wants to know that it’s communicating with its own server, not something just resembling it but in fact being a trap. The server also wants to get to know the other server better before it trusts it with a valuable message. Only after they check each other’s papers and establish some rules for cooperation, they’ll proceed with the transmission.

The handshake consists of several steps and it goes as follows:

The email client sends a “hello” message in which it states which SSL/TLS versions it’s compatible with and the encryption types it supports.
The server says ‘“hello” back but its response is different. It includes its TLS Digital Certificate and its public encryption key.
The email client verifies if the certificate is legitimate. If so, it proceeds with…
generating a Shared Secret Key with the server’s public encryption key it just received two steps ago.
The server then decrypts the received Shared Secret Key.
If everything went smoothly, both sites can now use this Shared Secret Key to encrypt and decrypt messages sent between each other.

If you’re interested in a much more detailed description of this encounter, check out this interesting breakdown of every byte used in a TLS handshake.

Notice how the encryption of this interaction was, at first, asynchronous (both sides were using different keys). Over the course of the handshake, they managed to establish that using Shared Secret Key was the way to go and the encryption became synchronous (and, as a result, faster).

Inspect Your Emails

Opportunistic vs Forced TLS. STARTTLS explained.

For a Handshake to happen in the first place, the connection between both sides needs to be established. TLS comes with the choice of two different approaches to establishing communication:

With Opportunistic (Explicit) TLS, an email client during a Handshake will tell an email server that it wants to talk but in private. In other words, it will suggest a change from a plain, not encrypted SMTP connection to a TLS-encrypted connection. If the attempt fails, however, it will start the transmission in plain text, without any encryption applied.

With Forced (Implicit) TLS, an email client will demand that they talk in private (and use the encrypted connection) An email server will then respond whether such an arrangement works for it or not. If it’s not compatible with the version of TLS used by the client, doesn’t support TLS at all or the connection fails, the transmission will be stopped and the email won’t be moved any further.

STARTTLS is the SMTP command used to initiate the switch to an encrypted connection when using Opportunistic TLS. Despite its name, it can also be used to initiate a switch to SSL if only this protocol is enabled for both sides.

Virtually all email clients that provide TLS encryption to their users offer Opportunistic TLS by default. Forced TLS is often possible to enable upon the user’s request but be sure to consider the pros and cons of such an approach first. The choice comes down to whether you want to ensure the maximum deliverability (Opportunistic TLS) or the maximum privacy (Forced TLS). With the first approach, you risk sending some (likely very few) emails without encryption. With the second one, your deliverability might suffer a bit. The choice is yours.

End-to-end encryption

Another approach to securing email communication is with end-to-end encryption. As we mentioned above, SSL/TLS only encrypt email on its way between the sender’s email client and his/her email server. A message is then decrypted and it travels all the way to the recipient’s inbox without any encryption, being vulnerable to attacks.

End-to-end encryption methods provide a different approach. A message is encrypted on a sender’s device before it’s even sent from a client. It then travels through the network (often being encrypted with TLS on the way, giving it extra security). When it arrives at the recipient’s client, it’s decrypted by a recipient. Throughout the course of the transfer, the message is encrypted at all times. However, to ensure proper implementation, a third-party pentest can help identify any vulnerabilities in your encryption scheme. Even if it falls a victim to hijacking, the data will be useless for a thief.

In order for end-to-end encryption to work, both sides need to be compatible with a given encryption method and have the right keys to encrypt/decrypt a message. Let’s say Charlie (a sender) would like to send a message to Mary (a recipient) using end-to-end encryption. Here’s how the process would look:

Mary generates a public key and a private key. From now on, she’ll be sharing the public key with anyone willing to message her but will keep her private key… private.
Charlie heard about Mary’s public key and he uses it to encrypt his message. The content of an email is turned into something called ciphertext, what looks like a random set of characters.
The message is sent from Charlie’s email client all the way to Mary’s client, following the standard procedures. If someone was to see the message on the way, they would only see a ciphertext.
The message arrives and Mary uses her private key to decrypt the message.

Mary realized that what Charlie so carefully encrypted were flight tickets for her surprise birthday trip. How exciting! Immediately afterward, she requested Charlie’s public key, wrote her response and encrypted it with the received key. He’ll know what to do, she thought.

Most popular end-to-end encryption methods

S/MIME

Secure/Multipurpose Internet Mail Extensions is a very popular encryption method. It relies on asynchronous encryption and again, a set of a public and a private key. As was the case with TLS, a sender willing to encrypt a message uses a recipient’s public key. When a message is received, a recipient utilizes his/her own private key to see the content of an email. S/MIME also allows you to add a digital signature to an email, certifying that you’re really a sender. This helps to prevent spoofing. This method can be easily added to popular email clients such as Gmail or Outlook.

PGP

Pretty Good Privacy (gotta love the name!) is probably the most popular encryption method and has been around since the early 90s. It’s not only used to encrypt emails but also files, directories, and entire disk partitions. Email-wise, the way it works is no different than S/MIME. It also relies on the combination of private and public keys but an encrypted data is even more difficult to crack. PGP uses the mix of data compression, public-key cryptography and hashing to achieve a nearly unbreakable string of characters representing the protected data. To validate the sender, their private key is also used to certify the ownership of an account. Since 1997, PGP is available as a non-proprietary standard called OpenPGP and everyone is free to implement it into their software.

Bitmessage

Bitmessage is often referred to as a “bitcoin of communications” and its users claim its much more secure and easy to use than the previous two methods we covered. To send messages this way, each user creates a Bitmessage address that consists of two keys – a public and a private one. As was the case in all cases above, a public key is used to encrypt messages and a private one to encrypt them. When decrypting a message, various additional processes are applied to certify that you are indeed a sender of a message. These can include hashing, data mapping, message “signing” as well as providing a “proof-of-work”.

Finally, a message is sent to a public network of thousands of users. Each member downloads every single message received but is able to decrypt only the ones encrypted with their public key. This keeps both the message body and metadata absolutely secure and virtually impossible to penetrate both for casual hackers and sophisticated government agencies. Needless to say, Bitmessage experienced explosive growth after a 2013 scandal that revealed the NSA’s email surveillance practices.

End-to-end encryption methods are certainly more secure than TLS and should be used when sending confidential data. While implementing Bitmessage to send weekly reports to your supervisor might be a bit of an overkill, both S/MIME and PGP are worth consideration. Always keep in mind that both sides need to be compatible with the method used, otherwise, the other side might not necessarily comprehend what you really had in mind.

Other considerations

Once you have TLS and/or end-to-end encryption set up, it’s also worth having a look at how to equip your emails with additional authentication methods. They’re used to prevent spoofing of your communication and also to improve the deliverability of your emails.

The most commonly used methods are SPF and DKIM.

SPF is used to identify a sender by uploading to the DNS records the IP addresses used to send emails on behalf of a given domain. The receiving client, before delivering a message, checks if the right address was used and might discard a message if it detects any abnormalities.

DKIM, on the other hand, is a digital certificate that’s sent with an email. It allows a recipient of a message to verify if the content of an email or headers were not modified (forged) during a transmission. Failed DKIM test affects the deliverability of a message and might be a reason for it to skip an inbox.

DMARC is the most sophisticated of all three methods and it leverages the other two methods to perform additional checks. It’s the only method that except for running a test, can also suggest to a receiving server what to do if a message fails a check.

It’s worth having all three set up for your domain. We cover them all in detail in separate articles, check them out at the links above. In case you want to know if yours are configured correctly, feel free to use our SPF, DKIM and DMARC record checkers.

Finally, it’s worth checking if your SMTP server works as expected. You can do it via Telnet or use a number of tools that offer such checks. See our article on testing SMTP server.

Once your SMTP server is checked, you still want to run tests on your emails before sending them out. The best way to do this is in a safe environment like Mailtrap Email Sandbox, which catches all of the outgoing SMTP traffic and removes the risk of spamming recipients. With its HTML check and spam score analysis features, Email Sandbox helps you inspect and debug emails in staging. Additionally, you can integrate your app with the Sandbox via API and perform automatic recurring email tests. Learn more about Email Sandbox in our Getting Started Guide.