Setting Up a Message Queue (Redis Streams) for DevOps Infrastructure

Message Queue on Redis Streams: Bottlenecks Under Load

A message queue on Redis Streams rarely breaks immediately. The first failure manifests at peak: a consumer fetches a batch of entries within a group and terminates midway — unacknowledged messages remain assigned to it.

Until another consumer rereads them after the idle timeout, these events are not processed at all, even though from the outside the queue looks calm.

The second bottleneck is duplicates. The delivery guarantee here is at least once: any acknowledgment timeout or redistribution of entries returns the message to processing again.

Without idempotency by entry ID, this produces double charges, repeated emails, and counters that diverge.

The third is processing lag. While consumers cannot keep up with the stream, the stream grows faster than it is drained, and the group lag stops being linear.

Adding consumers helps up to a certain limit: beyond that, processing hits acknowledgments, which themselves become a hotspot.

The fourth is message loss during peaks. If stream length is trimmed approximately, under load the trimming manages to evict entries not yet read.

The same result comes from acknowledging before actual processing and from a primary node failure without waiting for the write on the replica.

All four failures are reproducible and diagnosed by specific signs: the pending entries list, group lag, consumer idle time, and the share of reprocessed messages. Next, we break down how this is addressed at the configuration level.

What Changes in Service Operation After Implementing the Queue

A queue on Redis Streams moves event exchange from a "send and hope" mode to one with delivery acknowledgment.

Instead of direct calls between services, a buffer appears that survives a handler restart, and the load is distributed among multiple consumers within a group.

• Throughput. Reading in batches instead of single requests and parallel consumers in one group provide a steady processing rate: peaks are absorbed by the buffer rather than by service failures.
• Recovery time after a node failure. Unacknowledged entries remain in the pending list and after a restart move to a live consumer — recovery takes seconds, without manual triage.
• Share of lost events. Delivery acknowledgment and redelivery remove a whole class of failures: events do not disappear when the handler crashes and are not duplicated on resend.
• Service downtime. A consumer can be stopped for an update — events accumulate in the queue and are processed after startup; a separate approval window is no longer needed.
• Failure class. Instead of losing orders, reversible failures remain: processing lag, growing queue depth, retries. They are resolved without manual intervention.
• Observability. Queue depth, group lag, and the number of unacknowledged entries are visible in metrics — degradation is noticed before the user sees it, and checking "did everything arrive" stops being manual work.

Queue Deployment Options: Comparing Technical Approaches

The deployment mode determines not only fault tolerance but also how acknowledgments, redelivery, and journal growth behave. A single Redis Streams node holds data for one process — quick start, but a host reboot stops both publishing and processing.

A cluster and a managed service remove this dependency, adding constraints on the number of nodes and available commands.

Options should be compared along three axes: how quickly the queue recovers after a node failure, how much data it holds without memory degradation, and how much manual work operations require.

Switching between modes does not require rewriting consumer code: the connection address and retention policy change, while consumer groups and acknowledgments work the same way.

Therefore, starting on a single node does not close the path to a cluster — the mode is changed when a continuity requirement appears.

Implementing Redis Streams: Engineering Work Stages

We start Redis Streams implementation with an audit, not with configuration. First we need to see the actual data flows: who publishes events, who reads them, which queues are already running, and where messages are lost when a consumer fails.

Without these figures, the queue design turns into guesswork, and message lifetime and the number of groups are chosen at random.

1. Data flow audit. We compile a list of publishers and consumers, measure event frequency, and find places where database polling or synchronous calls currently operate instead of a queue.
2. Queue design. We determine the number of streams, message lifetime, consumer groups, and processing acknowledgment rules, as well as system behavior when a reader is unavailable.
3. Configuration. We deploy Redis Streams in a fault-tolerant configuration, set retention policies and memory limits, and enable consumer group lag metrics and alerts on growth of unprocessed messages.
4. Load testing. We run a peak stream, a consumer connection loss, and a node restart: we check whether processing recovers without event loss and how long it takes to drain the accumulated queue.
5. Stream migration. We migrate publishers and consumers one by one, keeping the old channel as a fallback, and compare the number of processed events on both environments.
6. Handover to operations. We provide documentation, queue diagrams, monitoring dashboards, and incident response procedures, and train the on-call shift.

Access Policies, Configurations, and Handover Procedures

A message queue on Redis Streams without documented rules quickly turns into a shared inbox: any service writes to any stream, and during incident analysis it is impossible to reconstruct who read what.

Therefore, along with configuration, we hand over a set of artifacts to the customer — for each one it is clear what is documented and how to verify it.

• Broker configuration — configuration files with memory limits, eviction policy, and timeouts; it is separately marked what can be changed on the fly and what requires a restart.
• Stream and consumer group diagram — a map: who writes, who reads, which group is responsible for reprocessing, and where unprocessed messages accumulate.
• Permission separation — a separate account for each service, with rights only to its own streams; service access is separated from administrative access.
• Access policies — which addresses and networks are allowed, whether channel encryption is enabled, and how credentials are issued and revoked.
• Documentation — topology, purpose of each stream, behavior on overflow, and the procedure for connecting a new consumer.
• Operations procedure — what the on-call engineer checks, alert thresholds, and actions when a node is unavailable or unprocessed messages grow.
• Backup and access rotation — what is backed up and how often, how to verify a working recovery point, and how to revoke keys.

We review the set together with the customer's engineers: we rehearse access revocation and recovery from a backup so that the procedure is tested in practice rather than merely written. After that, the team changes configuration and connects new consumers independently.

Case Study: Queue Under Peak Load Before and After Rebuilding

The project is a request processing service where peak load is several times higher than average.

The queue kept events in process memory: when a consumer restarted, unprocessed requests were lost irreversibly, and a stuck consumer did not return fetched messages — they were extracted manually from logs.

We moved processing to Redis Streams with consumer groups and acknowledgment: an event lives in the journal until it is acknowledged, and a failed consumer hands its messages to neighbors in the group.

The changes affected three areas. Acknowledgment removes an event only after successful processing, so a handler crash no longer means a lost request. Unacknowledged messages are returned to another consumer by timeout — the queue itself handles stuck ones.

The journal is limited by the number of recent events, memory does not grow indefinitely, and rollback to the previous set of handlers remains possible.

What to Do If the Queue Is Already Deployed and Running

Having a working queue on Redis Streams is not a reason to rewrite it from scratch. First we look at how it is designed: which streams exist, who reads them and at what rate, how many messages remain unacknowledged, and what happens on redelivery.

Most often the bottleneck is not in the stream storage itself, but in the absence of processing acknowledgment and length limiting.

Targeted improvements close this without downtime: we enable processing acknowledgment, set a maximum stream size with approximate trimming, handle stuck messages by timeout, and move them to a separate dead-letter stream.

Message format and stream names do not change — existing consumers continue to work without modifications, so migration as such is not needed.

If the logic requires rework, we bring up a parallel circuit: the new handler reads the same streams with its own consumer group, while the current one continues to operate.

We compare discrepancies using counters of processed and rejected messages; until parity is confirmed, the old circuit remains primary, and rollback amounts to shutting down the new one.

The phased migration proceeds one stream at a time: we switch the consumer group, monitor processing lag and stream depth, and only after a stable window do we remove the old code.

Zero-downtime operation is ensured by both versions reading from the same storage, while unacknowledged messages remain available to the new consumer.

The scope of work depends on the number of streams, consumers, and acceptable processing lag — this is what forms the migration plan.

Technical Doubts: Delays, Losses, and Scaling

Reliability doubts surface already at acceptance: the customer needs to be told what will happen if a consumer crashes or the stream grows severalfold. Below are answers to what is most often checked before putting a Redis Streams queue into production.

Are duplicates inevitable?

The "at least once" mode allows a repeat if the acknowledgment did not arrive. This is addressed by handler idempotency: the operation key is recorded in storage, and a repeated pass sees it and does not perform the action twice. Plus a retry counter and transfer to the problem queue after a specified number of retries.

Is message order preserved?

Strict order is maintained within a single stream. If it is needed per entity — order, account, device — we distribute events by entity key: within a stream the sequence is not violated, and different entities are processed in parallel.

How can we tell that the solution will withstand load growth?

Load testing runs on a profile that replicates production: peak stream, retry share, stopping one consumer. The results show where throughput stops growing — below that threshold we add consumers to the group, and the queue scales without rewriting code.

Let's Discuss the Task: Where We Start and What We Propose

To make the estimate fact-based rather than assumption-based, we start with input data. For Redis Streams, two things are critical: how events are born and how quickly consumers drain them — this determines stream configuration and where failures will appear.

• What the events are and who reads them: a list of streams, consumer groups, expected processing order, and acceptable lag.
• Load profile: average and peak frequency, message size, daily spikes — from this we derive the stream length limit and the eviction policy for old entries.
• Reliability requirements: whether message loss is acceptable, whether redelivery of stuck messages is needed, what counts as a final failure, and where such a message goes.
• Consumer logic: whether there is an operation identifier for duplicate protection, how handlers are designed, and how they survive a restart.
• Infrastructure and access: hosting location, disk persistence mode, redundancy, network restrictions, and maintenance windows.
• Observability: which metrics we consider key — growth of lag by group, share of unacknowledged messages — and where alerts should arrive.
• Existing constraints: security requirements, internal procedures, compatibility with the stack already in use.

In response, you receive a stage-by-stage scope of work, an estimate, and an implementation plan with responsibility areas and control points. Describe the task and send what you already have — we will get back with an estimate and work plan.