Amazon SQS/SNS: Message Queue Setup for DevOps
Is your support team complaining about duplicate orders, while developers report event loss after deployments? Chances are your message queue is misconfigured. As DevOps engineers with 7+ years of experience, we’ve solved this exact problem dozens of times. We’ll set up a reliable message exchange on Amazon SQS and SNS for you in 1–3 days. Contact us for an audit — we’ll identify queue issues within an hour.
Message queues in AWS solve two distinct problems that are often confused. SQS is a queue with a pull model: the consumer itself fetches messages, processes them, and deletes them. SNS is a pub/sub broker with push delivery: one publisher, many subscribers, real-time delivery. Choosing the wrong one is a source of reliability and cost issues.
How to Choose Between Standard and FIFO?
Standard Queue offers high throughput (millions of messages per second), at-least-once delivery (a message may be delivered twice), and no order guarantee. Suitable for most tasks where the consumer is idempotent. FIFO Queue guarantees exactly once, strictly in order, but maxes out at 3000 TPS (30000 TPS with batching). Use it for financial transactions, order processing, where order is critical. The queue name must end with .fifo. Standard Queue is 1000 times cheaper than FIFO under high loads — the difference in storage and operation costs is significant.
Comparison of characteristics:
| Property | Standard | FIFO |
|---|---|---|
| Throughput | ~millions msg/s | 3000 TPS (30000 with batching) |
| Delivery | At least once | Exactly once |
| Order | Not guaranteed | Strict order |
| Name suffix | Not required | .fifo |
| Scenario | Idempotent consumers | Critical order |
A common mistake is using FIFO everywhere "just in case." This limits throughput and increases costs. If order isn't needed, use Standard. In one project, a client had FIFO on all queues and got 500 TPS when they needed 2000 — after switching to Standard, throughput increased 10x and costs dropped 40%.
Why is DLQ Mandatory?
DLQ is a queue for messages that couldn’t be processed after N attempts. Without DLQ, a poison pill will keep returning to the queue forever, blocking processing. We always include DLQ with three attempts — that’s the production standard.
Configure the redrive policy:
{ "deadLetterTargetArn": "arn:aws:sqs:eu-west-1:123456789:myapp-dlq.fifo", "maxReceiveCount": 3 } After 3 failed attempts, the message moves to the DLQ. Set up a separate CloudWatch Alarm on ApproximateNumberOfMessagesVisible > 0 for the DLQ. If something appears in the DLQ, it must be investigated immediately.
Terraform example of creating a queue with DLQ
resource "aws_sqs_queue" "main" {
name = "myapp-orders.fifo"
fifo_queue = true
content_based_deduplication = true
visibility_timeout_seconds = 300
redrive_policy = jsonencode({
deadLetterTargetArn = aws_sqs_queue.dlq.arn
maxReceiveCount = 3
})
}
resource "aws_sqs_queue" "dlq" {
name = "myapp-orders-dlq.fifo"
fifo_queue = true
}visibility_timeout_seconds must be greater than the maximum message processing time. If Lambda timeout is 60 seconds, set visibility timeout to 90–120 seconds (100% headroom). Otherwise, Lambda is still processing, but SQS considers the message "not taken" and gives it to another consumer.
If your queue is already losing messages, order an audit — we’ll find the root cause within a day.
SNS Fan-Out Pattern
SNS + SQS together enable fan-out: one message is published to an SNS topic and delivered to multiple SQS queues simultaneously. Each consumer processes independently. Typical case: event order.created → SNS → three queues: billing-queue, inventory-queue, notification-queue. Three different services process independently, unaware of each other.
SNS supports filter policies — a subscriber receives only messages with specific attributes. For example, billing-queue gets only events with amount > 100, notification-queue gets all. This reduces load by 30–40%.
Integration with Lambda
Lambda Event Source Mapping for SQS is a built-in polling mechanism. The Lambda function automatically triggers when messages appear. Important parameters:
-
batch_size— how many messages Lambda receives per invocation (1–10000). For FIFO, we recommend small batches (10–50) — the entire batch is blocked if one message fails. -
maximum_batching_window_in_seconds— Lambda waits the specified time (up to 300 seconds) to gather a full batch. Saves invocations under low traffic. -
function_response_types: ["ReportBatchItemFailures"]— partial batch success. Lambda tells SQS which specific messages failed (not the whole batch). ReportBatchItemFailures is critical for FIFO. Without it, if one of 10 messages fails, all 10 go to retry.
Encryption and IAM
SQS supports SSE via AWS KMS (SQS-Managed Key or Customer Managed Key). For production, we use CMK — it provides an audit trail via CloudTrail. Strictly control SQS permissions via IAM: producer gets only sqs:SendMessage, consumer gets sqs:ReceiveMessage, sqs:DeleteMessage, sqs:GetQueueAttributes. No sqs:* in production. Additionally, configure resource-based policies for cross-account access.
Monitoring
Key CloudWatch metrics:
| Metric | What it signals |
|---|---|
ApproximateAgeOfOldestMessage |
Queue is not keeping up |
ApproximateNumberOfMessagesVisible |
Backlog accumulating |
NumberOfMessagesSent |
Sharp drop = problem with producer |
DLQ ApproximateNumberOfMessagesVisible > 0 |
There are broken messages |
Also add an alert on ApproximateAgeOfOldestMessage if it exceeds 5 minutes — this catches the problem before the queue grows to thousands of messages.
What You Get at the End
After completing the setup, we deliver:
- Terraform configuration files (modules for SQS, SNS, Lambda, CloudWatch).
- IAM policies with least privilege.
- Configured CloudWatch dashboards and alerts.
- Documentation on queue structure and integration patterns.
- Performance and cost recommendations (e.g., using reserved concurrency for Lambda, saving up to 30% on batch size).
To get a ready configuration, contact us.
Process
- Analysis — review current architecture, analyze queue issues (loss, duplication, delays).
- Design — choose queue types, routing schema, access policies.
- Implementation — write Terraform/IaC, configure integrations, test.
- Testing — load testing, idempotency and error handling checks.
- Deployment — deploy to production with a post-mortem plan.
Timelines
Basic setup of SQS + Lambda integration + DLQ + CloudWatch alerts — 1 day. Full configuration with SNS fan-out, filter policies, FIFO for ordered processing, KMS encryption, Terraform module — 2–3 days. We’ll provide an accurate estimate after auditing your infrastructure. Contact us to discuss details. Get a consultation and preliminary estimate.
Our engineers have 7+ years of AWS experience and have implemented 50+ message queue projects. Share your situation — we’ll assess your project and propose an optimal turnkey solution.
Amazon SQS Documentation: https://docs.aws.amazon.com/AWSSimpleQueueService/latest/SQSDeveloperGuide/welcome.html