All You Wanted to Know About Amazon SQS in 10 Minutes

When dealing with cloud-based infrastructure, it’s hard to pass by Amazon AWS. If you are considering Amazon SQS for improving your app architecture and need to quickly understand its basics and decide if it’s a fit for your project – you are in the right place. We will go over its basic functionality, main use cases, pricing, and common questions about this system and its integrations. 

How does Amazon SQS work?

Amazon Simple Queue Service (SQS) is a message queuing service that enables you to manage asynchronous communication between applications or microservices.

If you are new to queueing, imagine this process as a communication between two subscribers on a mobile network. There always should be at least two participants: the producer (the one who sends the message) and the consumer (the one who receives it). For direct communication (like a call) both participants must be online and available. 

Asynchronous communication is similar to messaging through a platform or even SMS. In this case, the producer sends a message and if the consumer is not available at the moment, this message will “wait” on an intermediary server. As soon as the consumer is available, the message is delivered and the consumer processes it (reads, forwards to another subscriber, etc.) 

With Amazon SQS, messages can be processed in two ways:

• Standard queues. This is the default type, which supports at-least-once message delivery. Your app should be able to handle messages arriving more than once and out of order. 
• FIFO queues. FIFO (first-in-first-out) model is used to process messages only once and in a strict order. 

Unlike the conversation, queued messages need to be deleted after processing to avoid their duplication. 

This is a very simplified description of how message queueing works in SQS. To dig deeper, check out the basic Amazon SQS architecture in the official documentation. 

Here are several important terms to understand before you start working with Amazon SQS:

• Amazon SQS Visibility Timeout. This is a period of time when a message is protected from receiving and processing by other consumers. You can configure it from 0 seconds to 12 hours.

• Dead-Letter Queues. They are used for messages that can’t be processed successfully for some reason. Failed messages will be moved to a dead-letter queue automatically, you just need to specify the conditions. Connect the source queue with a dead-letter one and set the maximum received count: the number of times a message failed to process. After the limit is reached, message is redriven to the dead-letter queue and removed from the source queue. Also, you can configure notifications for failed messages so that you can instantly check those in dead-letter queues and analyze what went wrong. Dead-letter queues are good to use as a starting point to examine the contents of failed messages as well as the logs of exceptions that caused them to fail. 

• Amazon SQS Short and Long Polling. These are two types of messages consuming. Long polling occurs when the waiting time for the ReceiveMessage API action is greater than 0. If the waiting time equals 0, then it’s short polling. Short polling is used by default. With it, you get a response back faster as it asks a subset of Amazon SQS servers (where the queue stores and processes messages), therefore you might need to issue multiple subsequent requests to get all messages from all servers. With the long polling, Amazon SQS waits for messages (until it times out) and asks all servers. In this case a request takes more time to be processed but there are higher chances that it won’t be empty.

How much does Amazon SQS cost?

Similar to other Amazon services, SQS is charged according to the “pay for what you use” model. In fact, you pay for the invoked requests (1 request = 64 KB chunk of a payload). The first one million requests per month are free. Each following million requests will cost you $0.4 if you use standard queues or $0.5 if you use FIFO. 

It’s important to note that a single API call can have from 1 to 10 messages and is limited to a maximum total payload of 256 KB (4 chunks of 64 KB). Also, you need to remember, that you are charged for the outbound data transfer. 

To be able to predict Amazon SQS cost, you should estimate the number of requests for each type of queue per month as well as the data transfer volume. 

To get all the pricing details and calculate your possible expenses, refer to Amazon SQS pricing. 

How to configure the Amazon SQS queue

You can access Amazon SQS via AWS Management Console or integrate it via API. 

AWS Management Console 

Getting started with the AWS Management Console is easy and straightforward. Just assess SQS form the Services list on your AWS account homepage and follow the instructions. You will get explanations and hints from Amazon at every step. 

Here you will be able to choose the type of the queue, make necessary configurations, and then start adding messages right away. Also, you can add metadata for each message, such as name, type, and value. 

Once your message is sent to a queue, it is ready for retrieval and processing by another application. To connect your app with the queue in this Amazon service, you will need to pass the queue URL (which you will find in the queue Details tab).

From the Queue Actions menu, you can request a message to view (by clicking  Start Polling for messages) and then delete it. 

Also, you can subscribe queue to the SNS topic and configure trigger from Lambda function right from the Queue Actions menu. 

For more details and further steps, refer to this guide. 

API integration

Amazon AWS provides SDKs for the most popular programming languages: C++, Go, Java, JavaScript, .Net, Node.js, PHP, Python, and Ruby. You can find them here. 

For SQS, the following automatic functionality is included:

• the cryptographic signing of your service requests
• retrying requests
• handling error responses

To integrate SQS with your app via API, you will need to construct an Amazon SQS endpoint first, then make GET and POST requests, and interpret responses. For detailed instructions, refer to the SQS Developer Guide. 

When to use Amazon SQS

Amazon offers three services to enable communication between apps: SQS, MQ (Message Broker), and SNS (Simple Notifications Service). 

Amazon MQ is a managed message broker service for Apache ActiveMQ. Amazon recommends using it for migrating your messaging with existing applications to the cloud. 

Amazon SNS is a push notifications service, and using it in combination with SQS is one of the best practices. 

Amazon SQS will be a good fit for new applications built in the cloud. You can use it independently but it’s always a good idea to compose it with other Amazon services like Amazon EC2, Amazon EC2 Container Service (Amazon ECS), and AWS Lambda. In addition, you can store SQS messaging data in Amazon Simple Storage Service (Amazon S3) and Amazon DynamoDB.

In general, message queueing is used to provide application scalability and to decouple the complex back-end operations from the front-end output. 

Here is what else you can do with the queues in Amazon SQS:

• enable server-side encryption 
• manage queue permissions
• use tags for a queue
• send messages with attributes and with a timer, etc.

And a couple of words about important Amazon SQS limits:

• You can retain messages in queues for 14 days maximum.
• The maximum size of one message is 256 KB (you can send large messages via Amazon SQS Extended Client Library for Java, which stores larger payloads in Amazon S3).
• A message can contain XML, JSON, and the unformatted text. The limited number of Unicode characters is supported as well.  
• The number of messages handled in standard queues is not limited! However, FIFO queues allow up to 3,000 messages per second.

How to test Amazon SQS?

Once you have configured communication between your apps with Amazon SQS, you need to make sure that everything works correctly. Amazon suggests integrating SQS with their AWS X-Ray service, which collects details about your requests and provides options for their further analysis and optimization. 

The setup doesn’t involve much effort. And what is also important, you can use in AWS X-Ray in almost no cost. It has a perpetual free tier with the first 100,000 traces recorded and the first 1,000,000 traces retrieved or scanned each month.

With the integration of AWS X-Ray with Amazon SQS, you can trace messages transferred through an Amazon SQS queue. Tracing is implemented with the help of trace headers. There are two of them supported in Amazon SQS: 

• default HTTP header (populated automatically by X-Ray SDK if Amazon SQS is called through the AWS SDK)
• AWSTraceHeader System Attribute – the user-specified message system attribute value in Amazon SQS. 

In Amazon SQS, you can send a trace header automatically through the SendMessageBatch or SendMessage call. You will then find the traced queues on the service map in the X-Ray console (as downstream nodes).  

Further, you need to retrieve the trace header and recover trace context. Those should be configured manually in three steps, as per Amazon documentation:

• Receive the message from the queue for the AWSTraceHeader attribute by calling the ReceiveMessage API.
• Retrieve the trace header from the attribute.
• Recover the trace ID from the header. Optionally, add more metrics to the segment.

Here is an example of how you can implement Amazon SQS testing with Node.js:

// https://docs.aws.amazon.com/AWSJavaScriptSDK/latest/AWS/SQS.html#receiveMessage-property
var params = {
  QueueUrl: "<QUEUE_URL>", // required
  AttributeNames: ["AWSTraceHeader"]
}
sqs.receiveMessage(params, function(err, data) {
  if (err) console.log(err, err.stack)
  else console.log(data)
  data.messages.forEach(function(message) {
    var headerString = message.Attributes["AWSTraceHeader"]
    if (headerString) {
      // comes from aws-xray-sdk-node
      var trace = AWSXRay.utils.processTraceData(headerString)
      trace.Root // trace id or null
      trace.Parent // id of the parent segment or subsegment or nil
      trace.Sampled // "0" means not sampled
    }
  })
})

And here is one more with Ruby:

# https://docs.aws.amazon.com/sdkforruby/api/Aws/SQS/Client.html#receive_message-instance_method
response = client.receive_message({
  queue_url: "<QUEUE_URL>", # required
  attribute_names: ["AWSTraceHeader"]
})
response.messages.each do |message|
  header_string = message.attributes["AWSTraceHeader"]
  # comes from aws-xray-sdk
  trace_header = XRay::TraceHeader.from_header_string(header_string)
  trace_header.root # trace id or nil
  trace_header.parent_id # id of the parent segment or subsegment or nil
  trace_header.sampled # 0 means not sampled
end

For details, refer to the AWS X-Ray Developer Guide. 

Further steps 

Amazon offers an exhaustive list of services for cloud computing. You definitely benefit from combining them, and Amazon SQS is not an exception: it works perfectly fine with Amazon SNS (a distributed publish-subscribe system) and AWS Lambda (serverless computing platform). Also, you can integrate Amazon SES, SNS, and SQS to manage email notifications you send to clients as well as handling their replies and/or bounces. 

Compared to other services, Amazon is scalable and reliable and offers the most competitive prices on the market.