Configuring AWS SQS / SNS for Queues and Notifications

Configuring AWS SQS / SNS for Queues and Notifications

When a message queue is misconfigured, data loss and duplication become routine. In one project, a Standard queue with VisibilityTimeout of 30 seconds processed videos that took 5 minutes — the message was re-read 10 times, clogging logs and tripling costs. We've been fixing such configurations for over five years: we design SQS and SNS so that the backlog doesn't grow and messages are delivered exactly once.

What Problems Do We Solve?

Standard or FIFO?

Standard Queue — at-least-once, order not guaranteed, throughput nearly unlimited. Suitable for email, image processing, indexing. Duplicates are possible — the consumer must be idempotent.

FIFO Queue — exactly-once, order within a message group. Throughput: 300 TPS without batching, 3000 TPS with batching (batch size 10). The name ends with .fifo. Requires MessageGroupId when sending. Don't use FIFO where Standard suffices — it imposes unnecessary constraints. In practice, Standard is 2x faster than FIFO under the same CPU load.

How to Avoid Duplicates with VisibilityTimeout?

After ReceiveMessage, the message is hidden for VisibilityTimeout (default 30s). If the consumer does not call DeleteMessage, it becomes visible for reprocessing. A typical mistake: timeout shorter than processing time. For example, video processing takes 5 minutes, timeout 30s — the task runs dozens of times in parallel. Rule: timeout ≥ max(processing time) * 1.5. For unpredictable tasks, use ChangeMessageVisibility to extend.

Long Polling (WaitTimeSeconds: 20) is mandatory. Short polling checks only a subset of servers — returns empty responses even when the queue is not empty. According to AWS documentation, Long Polling reduces API calls by 30–40%.

How to Configure a Dead-Letter Queue?

DLQ — a separate queue for messages after N failed attempts. maxReceiveCount = 3–5. Without DLQ, a broken message will cycle indefinitely, blocking FIFO or wasting resources. Dead-Letter Queue must be monitored: CloudWatch Alarm on ApproximateNumberOfMessagesVisible > 0 signals consumer problems. Get a consultation on DLQ setup from our engineers.

How Does Long Polling Work and Why Enable It?

Long Polling reduces latency and empty responses. With WaitTimeSeconds = 20, the request hangs for up to 20 seconds until a message appears. This lowers network load and API call costs. Enable via console or SDK — QueueAttributeName: 'ReceiveMessageWaitTimeSeconds'. Without Long Polling, empty responses can account for up to 90% of requests — wasted costs.

Fan-out via SNS: Broadcasting Events to Multiple Consumers

SNS topic → multiple SQS queues: one order.created event is processed in parallel by billing, notification, inventory. Each service operates at its own pace. Important: SQS Access Policy must allow sns:SendMessage. Message Filtering — a subscriber receives only relevant events, e.g., by region. This reduces load without extra code.

{
  "FilterPolicy": {
    "event_type": ["order.created", "order.cancelled"],
    "priority": [{"numeric": [">=", 5]}]
  }
}

Lambda as SQS Consumer

Event Source Mapping manages polling and scaling. Maximum batch size: 10000 for Standard, 10 for FIFO. ReportBatchItemFailures — always use it. When one message fails, only the failed ones are returned; the rest are not duplicated.

def handler(event, context):
    failed_items = []
    for record in event['Records']:
        try:
            process(record['body'])
        except Exception:
            failed_items.append({"itemIdentifier": record['messageId']})
    return {"batchItemFailures": failed_items}

Monitoring and Typical Metrics

Our Process

1. Analysis: study your current architecture, load, SLA. Identify bottlenecks: duplication, loss, delays.
2. Design: choose queue types, parameters (VisibilityTimeout, Long Polling), design DLQ and Fan-out.
3. Implementation: configure IAM policies, deploy via Terraform or CloudFormation, integrate with Lambda and EventBridge.
4. Testing: load tests, fault tolerance checks, error scenarios.
5. Deploy and monitor: deploy, set up CloudWatch alerts, dashboards.

What's Included

• Audit of current event architecture
• Design of SQS/SNS considering load and guarantees
• Configuration of DLQ, Long Polling, IAM policies
• Integration with Lambda, EventBridge, CloudWatch
• Monitoring and alerts (CloudWatch, SNS->Slack/PagerDuty)
• Documentation and team training
• 30-day post-deployment support

Timeline and Pricing

Basic SQS + SNS setup with Fan-out — 1–2 days. Complex event-driven architecture with filtering, DLQ, Lambda, and monitoring — 3–5 days. Pricing is determined individually — contact us for an estimate. Get a consultation and configuration within 1 day.

Our engineers have years of experience and AWS Solutions Architect certification. Incident SLA — 99.9%. Reach out to evaluate your project.