Yandex Message Queue: where asynchronous processing breaks
Under load, asynchronous processing breaks not in the queue but before it. While the order service uses synchronous calls to hit payments, warehouse, and email sending, response time is the sum of all participants — the slowest one sets the overall timeout.
During peak hours, the flow of requests grows severalfold, calls start failing by timeout, and the request disappears in the middle of the chain.
Then manual “catch-up” begins: an operator reconciles operation logs, looks for unconfirmed requests, and re-enters them. The higher the traffic, the more discrepancies between systems — some tasks are lost, some processed twice. Each such iteration is hours of engineering time and damaged customer trust.
A message queue such as Yandex Message Queue breaks this link: the sender publishes a task and immediately responds to the client, while consumers process it at their own pace.
Reliable message delivery rests on acknowledgements — if a handler crashes, an unprocessed message returns to the queue instead of disappearing.
Additionally, we configure retries, delayed processing of failed messages, and duplicate protection so that a retry does not create a second request.
Peak-traffic timeouts stop being lost requests — they turn into a delay of seconds. Below we discuss how this is addressed in practice.
What service decoupling provides and how to measure it
When a handler is called synchronously, the service response time equals the time of the slowest step, and its failure becomes the failure of the entire service.
A queue in Yandex Message Queue places a buffer between them, and each decoupling effect can then be measured in numbers.
Buffering peaks
The queue accepts messages faster than the handler processes them, so a spike several times above normal traffic is absorbed by accumulation rather than failure.
The share of tasks rejected due to overload drops to zero, while the front-end service maintains its previous response time.
Isolation of handler failure
An error in the consumer no longer brings down intake: messages remain in the queue, and the front-end service continues to respond. Recovery time comes down to restarting the handler — undelivered tasks are not lost and do not require manual replay.
Delivery retries
The number of attempts and the pause between them are set explicitly; after they are exhausted, the message goes to the dead-letter queue.
A failure in processing one type of message does not lead to their disappearance: the share of lost tasks is visible in metrics and remains manageable.
Independent scaling of handlers
The number of consumers changes separately from the front-end service: as queue length grows, handlers are added; as it drains, they are removed. Processing speeds up in proportion to load, while the front-end service is not rebuilt or re-released.
Standard and FIFO queues: what to choose
Delivery mode in Yandex Message Queue is chosen not out of habit but according to the flow pattern.
If events are independent and consumers process them in any order, there is no need to impose order: parallel handlers take as many messages as they can handle, and throughput grows with their number — just add consumers to the group.
Where order is part of the logic (status transitions, charges for a single object), a FIFO queue is needed.
It maintains sequence within a group of messages with a common key, but you pay for this with throughput: within a group, messages follow strictly one after another, and the number of groups and processing speed in each are limited.
| Variant | When it fits | Limitations |
|---|---|---|
| Standard queue | Independent events: mailings, logs, background jobs, processing order not important | Order not guaranteed, duplicate deliveries possible — the handler must be idempotent |
| FIFO with group key | Order matters within an object: order statuses, operations on a single account | Order only within a group, limits on the number of groups and speed in each |
| FIFO with a single stream | Strict sequence across the entire stream with a small volume | Low throughput, handlers do not scale horizontally |
The choice starts with calculation: how many messages per second, how many independent keys, what happens on repeated processing. If there are many keys and they do not intersect, FIFO mode provides almost the same throughput as standard. If there is a single key or retries are harmless, we take a standard queue and add handlers.
Setup stages: from flow diagram to acceptance
Work proceeds in fixed stages, and each ends with an artifact that can be checked before moving to the next — a configuration, diagram, or verification protocol. A failure at any step is visible before it reaches the production environment.
- Data exchange audit. We collect message producers and consumers, current transfer methods, volumes, and peak profiles. We separately record places where messages are lost or processed twice.
- Message flow diagram. We design routes: which event goes to which queue, what happens when a handler fails, where delivery order matters. The diagram is agreed before setup begins.
- Queue and permission setup. We create Yandex Message Queue queues for each flow, set message visibility timeout, retention time, and dead-letter queue. Access is granted to service accounts according to the minimum required set.
- Load testing. We run a profile close to peak: we monitor queue depth growth, the share of repeated processing, and system behavior when one of the consumers stops.
- Observability and alerts. We configure metrics for queue depth, delivery delay, and error count, and set notification thresholds so degradation is caught before user complaints.
- Handover to operations. We hand over the configuration, incident response regulations, and instructions for adding new flows; if necessary, we stay on duty during the first releases.
Configurations, diagrams, and regulations: what we hand over
We hand over, not screenshots of settings, but a set of artifacts for Yandex Message Queue: using it, the queue can be raised in a new environment, and an incident can be investigated without calling the author.
Everything is stored in the repository next to the code — changes go through review and reach the environment the same way as a service release.
- Queue configuration — descriptions of queues and subscriptions: queue type, message retention period, visibility window, batch size, timeouts. Values are derived from the load profile, not set by eye.
- Data flow diagram — producer, consumer, result route, reprocessing points. Places where the flow may be duplicated and how the consumer handles it are marked.
- Access policies and roles — which service accounts write and read, what exactly they are allowed to do, where keys with limited lifetime are. Permissions are separated to the minimum.
- Failed message handling — retry rules, threshold for moving to the dead-letter queue, procedure for investigating and returning to the flow after a fix.
- Metrics and alerts — signals about queue accumulation, growing retry share, and consumer failure, with thresholds at which the on-duty engineer receives a notification.
- Operations regulations — what the on-duty shift checks, how to edit configuration without losing messages, how to roll back a change, and what to do when a consumer fails.
Implementation case study: peak request flow without losses
What was there before the redesign
Requests were accepted by a synchronous service: the form sent the request directly to the handler, and the response to the client depended on whether the handler managed to process it.
While the flow was steady, the processing bottleneck did not show — but on promotion and mailing days, the number of requests jumped severalfold, the handler could not keep up, and some requests were lost at the intake: the user saw an error and did not retry sending.
On peak days, the share of lost requests reached 3–4%, and they were handled manually — logs were exported and missing entries were added.
What was redesigned
A queue based on Yandex Cloud was placed between intake and processing. The receiver only records the request and confirms intake, while handlers take messages at their own pace.
Acknowledgement goes to the queue only after successful completion: if the handler crashes, the message returns and will be processed again.
For requests that do not pass after several attempts, a separate processing queue was created so that a failed message does not block the others. Handlers scale by queue length, and the request key protects against duplicates.
What became
The share of lost requests went to zero within the redelivery window: a request does not “burn out,” even if the service restarts at the very peak. The accumulated flow of tasks, which previously took almost a full working day to process, now clears in minutes.
Manual interventions decreased noticeably — instead of night shifts for sorting out missing items, only isolated messages from the processing queue remain.
What to do if exchange between services already works directly
Direct service invocation works as long as a recipient failure is immediately visible: the producer holds the connection and retries the request. The same property turns overload of one section into failure of the entire chain.
A queue breaks the link: the message is accepted and stored until the recipient is ready to process it.
There is no need to switch to asynchronous exchange entirely at once. In Yandex Cloud, the first sections to move are those where a response is not required in the same session: events, notifications, exports, reference data recalculation. Requests on which the user interface depends remain direct.
| Section | Pattern | If recipient fails |
|---|---|---|
| Events, notifications, exports | Queue | Message remains in queue until recovery |
| Requests for interface | Direct call | Producer sees the error and returns it to the user |
| Batch recalculation, reports | Queue | Task is not lost, deadlines shift |
| Operations with single processing | Queue with acknowledgement | Redelivery is suppressed by the operation key |
Two patterns coexist at the same time thanks to routing by operation type: the producer decides where to send the call, and the fork is described in configuration rather than spread across code.
For the queue, retries, delayed delivery, and a dead-letter queue are configured — direct invocation does not provide this at all.
It is convenient to start with one receiving service and one consumer group — then you can see the real queue depth and processing time. Rollback is cheap: a switch on the producer side returns direct invocation, and the old branch works until the end of the transition period.
Limitations, quotas, and typical asynchronous processing errors
A queue is not infinite: there is a limit on the number of messages and volume, and each message has a retention time, after which it is deleted regardless of whether the consumer processed it.
Too short retention means messages disappear in the middle of a task, and events do not reach the consumer. During a load spike, the queue hits the limit, publication is rejected — data is lost at intake.
The second error is relying on single delivery. Yandex Message Queue provides at-least-once guarantee: when a consumer restarts or the visibility window is exceeded, the message arrives again.
Without idempotent processing, this produces duplicate operations — double charges, repeated emails, mismatched counters.
The visibility window is a key trade-off: a short one creates duplicates, a long one creates stuck tasks that are not reprocessed for hours.
Worst of all is when no one monitors the queue: monitoring depth and the age of the oldest message shows that consumers are falling behind or failing long before user complaints.
And a separate queue for messages not processed after several attempts — otherwise they loop or disappear without a trace.
How to distinguish overflow due to incorrect retention time from a normal shortage of consumers?
Compare retention time with the actual processing duration from metrics and look at the age of the oldest message: if it is small and the queue is growing, consumers are falling behind and new handlers are needed.
If messages disappear before being processed, retention is set too short. Set it with a margin for peak task duration, and move “tails” to a separate dead-letter queue.
Let's discuss the task: where to start and what we need from you
You do not need a multi-volume document to define the task — a description of flows, message examples, and reliability requirements are enough. From them, it is already clear which queue mode fits, how many consumers to keep, and where retries are needed.
- Description of data flows: who publishes messages, who reads them, what is considered successful processing, and where the result goes.
- Message examples: body structure, size, frequency, and peak hours — from them we calculate queue throughput.
- Ordering requirements: whether sequence within a group matters or processing is independent.
- Reliability requirements: acceptable delivery delay, reasonable number of retries, and rules for processing failed messages.