Kafka monitoring: where queues lose messages
Consumer lag behind the stream rarely looks like an outage: the broker accepts records, the availability metric is green. The reader's offset slowly drifts away from the end of the log, and after a few hours it is already hours of delay.
Individual partitions of the queue stand still, some messages go unprocessed and expire by retention period — and you learn about this from complaints, not from the monitoring system.
The reason is that the metrics set usually looks at the broker, not at processing. Average lag across the group smooths the picture: one stuck partition out of twenty gives a modest figure in the report, although it is exactly where unread messages accumulate.
Kafka monitoring works at the level of "consumer — partition — offset", otherwise lag growth remains unnoticed until the moment when the stream can no longer be caught up.
Silent processing failure is the most unpleasant scenario. The consumer continues to read and commit offsets, but the logic fails on a certain type of record: some messages go to the retry branch, some are discarded without a trace.
Externally the service works, but data is lost where no one counts it. Message loss stops being noticeable if there is no reconciliation of the number of written and processed events for each partition.
Therefore we build monitoring around what can be counted: lag in units of time, not only in the number of records, custom thresholds for partitions with different loads, signals for partition idle time and counter divergence, tracing the path from source to result.
Then lag growth is visible within minutes, stuck partitions are not masked by the average, and unprocessed messages are investigated before the retention period expires.
What end-to-end control of message queues provides
End-to-end queue control brings publisher services, the broker, and consumer groups into one picture: lag of each group, publication and acknowledgment rate, volume of unread partitions, share of redeliveries.
While these metrics are absent, a processing failure is visible only by an external symptom — the user did not receive an email, the order is stuck in an intermediate status.
Incident response time
Threshold alerts on consumer lag and acknowledgment rate move detection from "customer reported" mode to automatic: the on-call engineer sees sustained queue growth at an early stage, when the consumer is still catching up with the stream, not after processing has stopped.
Incident detection time drops from hours to a few minutes — you can intervene before the queue hits message retention limits.
A class of failures that stops reaching the user
We separately track messages without acknowledgment, growth in redeliveries, and "poison" messages that the consumer cannot process and endlessly returns to the queue.
Such failures are dangerous because they do not bring the service down but silently lose data: some operations simply do not complete, and this is discovered through complaints, not metrics.
Transparency of stream processing
Tracing links a message identifier to a specific business operation, so investigation is based on facts: you can see at which step the message was delayed and how many times it was reprocessed.
Kafka and other brokers expose these metrics natively — the task is to correlate them with consumer state and not drown in alert noise.
Three approaches to metrics collection: agent, collector, plugin
Queue metrics in Kafka and other brokers are collected in three ways, and they do not replace each other. The choice determines whether you will see consumer lag before the queue starts growing and how many resources the monitoring itself will consume.
What each option provides and what it lacks is in the table.
| Option | When it fits | Limitations |
|---|---|---|
| Node agent | Need system metrics alongside the broker and a unified format for all services in the environment | Consumes node resources, loses some measurements when the broker restarts, requires access to the machine |
| External metrics collector | Many heterogeneous environments, nothing can be installed on hosts | Polls over the network and at a lower frequency, does not see the broker's internal state |
| Broker plugin | Need metrics from the inside: queue depth, consumer lag, unacknowledged messages | Tied to the broker version, updated with it, monitoring disappears during a failure |
What to choose under high load
Under high load, monitoring overhead stops being invisible. External polling cannot keep up with the growth in the number of queues and gives a fragmented picture, so the plugin is made the primary method: it exposes metrics from the inside and does not traverse the network. The node agent remains for system metrics that the plugin does not see.
Mixed collection in one environment
Keeping two methods in one environment is normal if areas of responsibility are separated in advance: the plugin for queues and consumers, the agent for the node, the collector for availability and aggregate thresholds.
It is important not to count the same value twice, otherwise the graphs diverge and the on-call engineer stops trusting them. Broker and plugin upgrades are planned together so as not to be left without monitoring at the most inconvenient moment.
How we implement queue monitoring: work stages
We start with analyzing the queue environment: which brokers and queues are actually used, who publishes and who reads, where the critical processing path runs. Then we record deviations that have already led to incidents — lag growth, consumer stuck, disk full on a node. This determines what goes into the metrics scheme and what remains noise.
- Queue environment analysis. A map of brokers, exchanges, queues, consumer groups, and connections between services. We separately mark flows whose idling stops the process.
- Metrics scheme. A set of indicators: consumer lag, queue depth, publication and processing rate, number of unacknowledged messages, state of nodes and disk.
- Collection setup. Collection from brokers and consumers, binding to the metrics storage and dashboards. We label indicators by services and queues — the on-call engineer immediately sees the source.
- Threshold rules. Warning and critical for each indicator, linked to the on-call schedule. We take the threshold from observed behavior, not from a reference book, otherwise we get a stream of false positives.
- Testing with a simulated failure. We stop the consumer, inflate the queue, speed up publication: did the rules fire and did the notification reach the right person. We make corrections before handover.
- Handover to operations. Dashboards, rules, instructions for the on-call engineer, and the procedure for refinement when new queues appear.
What we need from your side at each step
Access to brokers, metrics storage, and dashboards, a list of critical queues and services, an on-call contact, and a window for safely simulating a failure. During analysis, we need an engineer who knows the service connections; at acceptance, the person who will be on call. Otherwise timelines go into approvals, and the rules remain untested.
What remains with you after implementation: diagrams and policies
Monitoring implementation ends not with a set of graphs but with a set of artifacts that your team uses without us. Everything lives in your repository and your alerting system — with comments explaining why it was done this way.
- Map of queues and consumers. Which topics and queues exist, which services read them, which consumer groups exist, and what lag is considered normal for each. With it, a new engineer understands the message exchange scheme in an evening.
- Metrics collection configurations. Described as code: which counters we collect from brokers and consumers, at what interval, where data is stored, and how long it lives.
- Alerting rules. Thresholds for consumer lag, queue growth rate, node unavailability, and publication failures — so that they fire on a problem, not on noise.
- On-call policy. Who is responsible and during which hours, what counts as a night incident, how escalation works, and when the second line is engaged.
- Response instructions. Step-by-step actions for typical failures: lag growth, node loss, disk full, stuck consumer — with diagnostic commands and criteria for "fix now or observe".
- Monitoring dashboard description. What to look at first when an alert fires and which metrics confirm that the problem is gone.
- Development plan. What to add next: end-to-end message path control, retention periods, alerts by business indicators.
How we transfer knowledge to your team
We transfer not as files on disk but through working walkthroughs: we go through each alert and explain why the threshold is what it is and what will change if it is moved.
We separately conduct drills — simulating lag growth and node loss so that the on-call engineer walks through the scenario in a calm setting, not during an outage.
Your team takes the first on-call shifts together with our engineer, then on its own, and the documentation remains the entry point for new people.
Case study: how message loss was eliminated during peak loads
A retail service with nightly order export: the flow into the queue during sales exceeded processing speed, buffers overflowed, and some messages disappeared before anyone noticed.
Kafka itself does not lose data — losses occurred at the joints: acknowledgment was sent before actual processing, and no one tracked consumer lag.
The failure was learned about from support — customers wrote that the order was not being assembled. From detection to the start of investigation, hours passed, and the investigation itself was manual: events were reconstructed from logs by reconciling order numbers.
What exactly was changed in the processing environment
Acknowledgment began to be sent strictly after the result was written; processing was made repeatable with ID checking — retries no longer turn into duplicate operations. Messages that fail processing after several attempts go to a separate queue with the body and failure reason.
Alerts were tied to two signals: consumer group lag and its growth rate — instead of a single threshold that fired on planned spikes. Monitoring was built on end-to-end markers: once a minute, a test message travels the entire path and confirms that the environment is alive.
| Metric | Before | After |
|---|---|---|
| Lost messages at peak | up to 2% during sales hours | a few, all enter the investigation queue |
| Failure detection | hours, from customer reports | minutes, from the lag alert |
| False positives | several per week | isolated per quarter |
| Manual investigation | after every peak night | scheduled reconciliation once a week |
How does Kafka monitoring differ from RabbitMQ monitoring?
Kafka is a distributed log, not a queue: messages are stored in partitions for a specified time regardless of whether anyone has read them.
Therefore Kafka monitoring is built around consumer lag — the difference between the last written and the last committed offset in the group.
They look not only at average lag but also at uniformity: if one consumer in the group has taken more partitions, local delay grows while the overall picture appears healthy. Load balancing across partitions matters more here than an absolute threshold.
RabbitMQ works differently: a message disappears after acknowledgment, so RabbitMQ metrics describe the queue here and now.
The key ones are queue depth, the ratio of publication and acknowledgment rates, the number of unacknowledged messages, and the presence of queues without active consumers.
These sets must not be confused. For a streaming bus, lag fluctuates during the day, and the threshold is set on growth rate, not on the absolute value: the log does not empty by itself.
For a queue broker, depth is a direct sign of failure: it either stays near zero or grows until it hits limits, after which publication starts failing.
In a mixed queue environment where both types of brokers coexist, the monitoring dashboard is assembled according to common rules, while thresholds and forecasts are calculated separately.
For a streaming bus, the forecast is the time until data is evicted by retention period; for a queue broker, it is the time until overflow and the risk of disk growth. The alert fires on a sustained trend, not on a single spike.
Technical questions asked before kickoff
Before kickoff, the customer's technical team usually asks the same questions — and rightly so: the answers determine how collection will be structured and what will go into alerts.
Below are five that we close before work begins, so we do not have to clarify them during the process.
How will you get access to the metrics?
We create a separate read-only account with rights exclusively to broker metrics — without access to the contents of topics and queues. Collection runs from inside your environment; nothing is published externally.
How much does collection load the cluster?
The agent collects counters through the broker's native interface and aggregates them locally, so brokers bear a fraction of a percent of their normal workload. Consumer lag, queue depth, and processing rate are still fully visible.
What about data security in the queues?
We read metadata and counters, not message bodies — the contents of communication between services remain inaccessible to the monitoring system. Only queue names and numeric values go into logs and alerts; encryption keys do not leave your environment.
We already have monitoring — will there be a conflict?
First we look at what is already being collected and connect to the existing collection rather than deploying a second one in parallel.
Usually it is not collection that is missing but thresholds and rules: we add the missing signals and configure alerts on top of the current picture.
Where does your responsibility end?
We are responsible for metrics collection, alerting rules, incident investigation, and documentation for them. Broker configuration, access, and the state of the cluster itself remain with your team — we record the boundary in writing before kickoff.
Let's discuss your task and come back with a work plan
To make the analysis substantive rather than a general overview, send three things: a diagram of the environment with brokers and consumers, the composition of queues and topics with their purpose, and current complaints — what exactly is slow, lost, or duplicated.
Exports of metrics for a week and how alerts are currently configured are also useful. From this set, the bottleneck is already visible, and we do not have to ask general questions about Kafka and RabbitMQ.
Environment readiness checklist that we use before work begins:
- service owners are named — with whom to coordinate restarts and consumer group changes;
- there is access to broker metrics and the lag of each consumer group;
- the expected mode is recorded: peak load, acceptable lag, retry policy;
- current signals are described: who receives alerts and at what thresholds;
- there is a test environment where a failure can be safely reproduced and rollback can be checked;
- the release frequency and windows when queue restructuring will not affect anyone are clear.
This way we will discuss the task using your data and come back with an estimate and work plan: stages, timelines, areas of responsibility, control points. We collect questions for the contractor together with the materials — we answer substantively, without general promises.
Work start does not wait for complete clarity on all points: we close the checklist as we go, and we provide the first observations already at the analysis stage.