AWS SQS and Decoupled Architectures: Benefits and Use Cases

AWS Simple Queue Service (SQS) and Decoupled Architectures

Benefits and Use Case Examples

Simple Queue Service (SQS) is a distributed message queue service in AWS that enables you to decouple the components of a cloud application. By using SQS, you can send, store, and receive messages between software components at any volume, without losing messages or requiring other components to be available.

Benefits of using SQS

Using SQS in your AWS architecture helps ensure that your application can continue to run and scale even if individual components fail. Benefits include:

Scalability

SQS automatically scales as your needs change, so you don't have to worry about capacity planning or manual administration.

Reliability

SQS provides at-least-once delivery of messages, meaning that messages are delivered once and duplicates are deleted automatically (for FIFO queues), even in the event of a failure.

Flexibility

SQS integrates with other AWS services, making it easy to add message-based workflows to your application.

Cost-Effectiveness

SQS is priced based on the number of requests and the amount of data you transfer, so you only pay for what you use.

Maintainability

Testability

By encapsulating parts of your infrastructure with SQS, you can test your infrastructure by sending specific messages to test input/output responses.

SQS Decoupling: Use Case 1

Using a queue is a great way to decouple segments of your infrastructure.  This serverless architecture receives a request & saves the data:

The API gateway to Lambda paradigm has a good degree of scalability and is adequate for many low traffic use cases. But you may start to see significant timeouts as you reach 1000 concurrent requests.  Let’s make this architecture more resilient by placing a queue in front of the Save Data lambda and utilize a pattern called queue based leveling. This acts as a buffer to manage traffic spikes so that downstream services can handle the load at the desired levels.

Adding a queue between API gateway and your workers allows you scale and handle spikes in your traffic. This is an easy way to dramatically increase your load capabilities. You can even batch messages to a single lambda to further reduce cost. Using a queue also allows multiple subscribers to messages. You can use the same message to trigger multiple asynchronous workers or use different messages to trigger different workers as seen below.

SQS Decoupling: Use Case 2

Let’s look at a different use case. You have an API which saves some data, processes that data for 30 seconds, then sends an email confirmation to the user. Your maximum load is only 100 requests per minute, so you aren’t worried about lambda concurrency issues.

There are potential problems if errors occur when processing the data. Perhaps you discover that data isn’t being processed correctly and you want to pause all requests until you’ve updated the code. To handle this, add a queue in front of the Process Data lambda.

Now that we have a queue in place, we can send a response back to the user once our data is saved and store the message which triggers processing of the data. You can store messages for the duration of your retention period for up to 14 days. When the Process Data lambda is fixed, you can release those messages held in your queue and process the data.

Planning for Growth? Implement SQS Early.

Overall, SQS can help you build a more scalable, reliable, and flexible application in AWS, while reducing the complexity and cost of managing message queues. Using a queue strategically in your architecture can save you from future customer complaints and downtime as your application grows. I recommend implementing queues early in your application, as they can solve many common architectural and SDLC problems that applications face as they grow. A good architect is constantly thinking about the growth of the application – and SQS is a great tool to have in your toolbox for preparing for the unexpected.

Bill Baker

DevIQ, March, 2023