Why Google Pub/Sub loses messages under peak load
Message loss rarely looks like an error in the logs — more often it is silence: the handler is running, metrics are steady, and some data simply does not arrive.
A subscription stores a message for a limited time and does not wait indefinitely: anything not acknowledged in time is deleted without being returned to the queue.
The first bottleneck is subscription overflow. Under peak load, publishing outpaces processing: unacknowledged messages and retries accumulate, the pending counter grows, and the handler receives more than it can accept. Messages do not sit in the queue forever — the tail of the burst is evicted before it is reached.
The second is acknowledgment timeout. The handler takes a message, goes into an external call or heavy query, and fails to acknowledge it within the allotted window.
The service considers the delivery unsuccessful and sends a copy: duplicates and delivery count grow, and in the absence of a retry policy the message goes to the dead-letter queue or is silently erased.
The third scenario is "stuck" handlers: the process is alive, metrics are normal, but acknowledgment is not sent — context is lost, the thread is blocked, the container goes into restart.
The message goes in circles until the retention period expires and is lost exactly where diagnostics were needed.
All three scenarios converge at one point — the configuration of the subscription, dead-letter messages, and access rights to them. We break down exactly where the flow breaks and how it is closed.
Measurable effect: fewer losses, duplicates, and downtime
Subscription configuration changes what is visible in metrics. Before it, a handler failure looks like silence: the message is delivered, acknowledged, the handler crashes — and there is no event either in the system or in the queue.
After configuration, anything that fails processing after a specified number of attempts goes to dead-letter: it remains in the triage queue rather than disappearing.
Duplicates are the second category of losses.
An extended acknowledgment window together with idempotency by event key eliminates reprocessing where it is not needed, and the remaining retries become safe: redelivery does not create a second record and does not perform a second charge. The volume of reprocessing falls because some failures are closed on the subscription side.
The third effect is recovery time. An incident is detected not by a customer complaint but by an alert on the age of the oldest unacknowledged message and on dead-letter growth: the on-call engineer sees the problem at the beginning, not at the end. Return to work is switching the subscription to a backup handler; this takes minutes, not hours.
A whole class of incidents goes away: "silent event loss," "double processing," "the queue grew for a day and no one noticed." Flow observability is built on three signals — age of unacknowledged messages, number of deliveries per message, dead-letter volume — and each is tied to a specific on-call action.
We work in Google Cloud Pub/Sub infrastructure: retry policies, access rights, and alert thresholds are defined declaratively and versioned together with the service code.
Google Pub/Sub modes: subscriptions, acknowledgments, retries
In Google Pub/Sub, the delivery mode seems like a detail while there is only one stream.
As soon as there are several consumers, it determines which failures you will get: a push subscription comes to your address and requires a quick response, while a pull subscription fetches messages itself, so the rate, batch size, and processing window are selected to match the load.
Ordering and acknowledgment are solved separately. An ordering key binds messages to one queue — order is maintained within the key, but parallelism drops.
Acknowledgment processing extends the window: if the consumer does not make it, the message returns to delivery, and after a specified number of attempts it goes to the dead-letter queue.
| Option | When it fits | Limitations |
|---|---|---|
| Pull subscription | own services, batch fetching, high rate | requires a polling loop and its own acknowledgment logic |
| Push subscription | events to external services, functions without a permanent consumer | requires an accessible address and a quick response; you do not control the acknowledgment window |
| Streaming processing | continuous stream with minimal latency | permanent connection, harder to pause and roll back |
| Multiple subscribers | one stream for different teams and scenarios | each has its own retry and dead-letter policies; duplicates are suppressed by idempotency |
| Ordering key | order within one entity matters | less parallelism; a hot key becomes a bottleneck |
The delivery mode determines the class of failures that you later handle on call. Below is how we connect subscriptions, rights, and monitoring into one managed scheme.
Implementation order: from flow audit to handover of regulations
We structure the order of work so that each step ends with a verifiable artifact — a diagram, configuration, or verification report.
We start not with settings but with a flow audit: an error in the message exchange scheme is easier to catch on paper than in production under peak load.
- Flow audit. Inventory of Pub/Sub topics and subscriptions: who publishes, who reads, what latency is acceptable, where errors do not surface in metrics.
- Scheme design. We arrange flows by criticality, fix the topology of topics and subscriptions, retention period, and a separate dead-letter topic for unprocessed messages.
- Retry configuration. Retry policy with delay between attempts, number of deliveries before sending to dead-letter, behavior on acknowledgment revocation: a failed message goes aside, and the flow continues.
- Access. Service accounts and roles with the minimum necessary set of rights: a subscriber does not publish, a metrics collector does not read message bodies.
- Failure testing. We disable a subscriber, revoke rights, break the handler — we look at what happens to the queue, whether retries fire, whether the incident is visible in monitoring.
- Load testing. We run a profile close to peak: how much the stream sustains without latency growth, how the volume of unprocessed messages grows, at what rate mass retries begin.
- Monitoring and handover of regulations. Alerts on the age of the oldest unacknowledged message, the share of messages in dead-letter, and publication errors; instructions for the on-call shift and the incident triage procedure.
Delivery package: flow diagrams, access policies, regulations
The delivery package is not a presentation but working documentation according to which your on-call shift and engineers act without us.
We record all Pub/Sub settings in your repository: diagrams, policies, and regulations are versioned, changes go through review, and do not live in someone's memory or chat.
- Flow diagram: topology of topics and subscriptions, who publishes and who reads, which messages are considered critical and what route they take during retries.
- Environment separation: test, pre-production, and production contours live in separate projects with independent rights and quotas — a load test cannot read production data.
- Access policies: roles at the level of specific topics and subscriptions, a separate account for each service, the principle of least privilege instead of one shared key for everyone.
- Alert thresholds: growth in the number of unprocessed messages, the age of the oldest of them, the share of delivery errors, and publication latency — with the norm and condition for calling the on-call engineer specified.
- On-call instructions: step-by-step scenarios for each type of failure — what to look at first, what is safe to restart, when to escalate, and to whom.
- Retry regulation: how many times and at what interval delivery is retried, at what number of failures a message goes to a separate queue, and how to triage it.
- Ownership map and change procedure: who is responsible for a topic, through what process a policy edit goes, and how to roll back an unsuccessful change in one step.
Case: event loss in a settlement loop — dynamics before and after
A stream of settlement events went through a bus without separating ingestion and processing: the subscriber acknowledged the message immediately after receipt, and business logic ran afterward.
An external module error or timeout meant the event was considered delivered and disappeared — a reconciliation discrepancy surfaced only the next day, and the cause was searched manually in logs.
We rebuilt acknowledgments: ingestion is recorded separately from the processing result, failed attempts go to redelivery with increasing delay, and those that do not pass within the allotted number of attempts go to the dead-letter queue with the original body and headers preserved.
Service access to topics and subscriptions was separated by individual roles — the loader account can no longer read and delete other people's subscriptions.
Dynamics by metrics
| Metric | Before work | After observation |
|---|---|---|
| Share of lost settlement events | about 2% per day | below 0.05% |
| Detection of processing failure | at morning reconciliation | by alert, minutes |
| Manual incident triage | triage of the entire log batch | by message ID |
| Rollback to previous processing scheme | loop restart | subscriber version switch |
The loop runs on Google Cloud Pub/Sub, and the key was not enabling retries itself but aligning the dead-letter message retention window with the on-call shift response time: if it is shorter, messages are lost already on the way out of the queue.
Over a month of observation, not a single settlement day closed with a discrepancy requiring manual adjustment of entries.
How a dead-letter message differs from a retried one
The boundary is not based on the fact of failed processing but on the attempt counter. As long as the attempt limit is not exhausted, the message remains in the subscription and is returned to the subscriber again — this is the standard redelivery mode, not an error.
The platform considers a message "in flight" until acknowledgment, and if acknowledgment does not arrive within the allotted window, it returns it to the delivery queue with an incremented attempt number.
The retry policy defines exactly how this happens: the interval between attempts and its growth from attempt to attempt.
Increasing delay is needed so that a failing service does not receive a stream of tasks faster than it can recover, and neighboring subscriptions do not clog the shared resource.
As long as the error is temporary — dependency unavailability, network hiccup, load spike — a few cycles are enough, and the message goes into processing.
Poison messages behave differently: the handler crashes on every retry for the same reason. Invalid schema, reference to a deleted object, violation of a data invariant — such failures are deterministic, and increasing attempts is pointless. There is exactly one sign: the same error class across different attempts.
When the counter reaches the limit, the platform must choose: delete the message or send it to a separate triage queue. The triage queue is not an archive or a dump but the on-call engineer's workplace: only what requires a manual decision or data correction goes there. If it is not set up, the platform will discard the message, and no trace will remain.
Practical conclusion: redelivery cures temporary failures, the triage queue preserves the irreparable automatically. Configure the retention period and notifications in the triage queue — otherwise it will itself become delayed data loss.
Technical concerns: delays, duplicates, access separation
Delivery delays are not a failure but a property of asynchronous exchange. In Google Cloud Pub/Sub, the path from publication to processing depends on batch size, acknowledgment window, and the speed of the handler itself.
If the subscriber responds longer than the window expires, the broker considers the message unprocessed and delivers it again — this is how duplicates appear.
We build the scheme so that a retry is safe: idempotent operations, a deduplication key inside the message, a separate queue for what could not be triaged within a specified number of attempts.
We distribute rights by service accounts — a publisher only publishes, a subscriber only reads and acknowledges, we do not grant administrative roles to applications.
We attach alerts to queue depth and the age of the oldest unacknowledged message, not to the error counter in logs: it is these metrics that show resource consumption and processing lag.
Messages arrive repeatedly — is this a configuration error?
No, the base guarantee is at-least-once delivery. Duplicates are removed on the handler side: a deduplication key and state check before writing. We enable ordering by key where order matters, but it does not replace idempotency.
How do you tell whether the issue is rights, not code?
A publication denial and an empty subscriber fetch look similar. We distinguish them by the access log: separate roles for the topic and for the subscription immediately show where exactly the denial is. We grant access on a least-privilege basis, so it does not have to be expanded for debugging.
Do settings affect resource consumption?
Yes. Batch size, acknowledgment window, and dead-letter retention period directly determine traffic and storage volume. We select them for the load profile rather than leaving defaults, and we back them up with alerts on queue growth.
Let's discuss your message exchange scheme and failure points
To come back with an estimate and a plan rather than generalities, we need your current message exchange picture.
A dump of topic and subscription configuration, a diagram of flows between services, and a description of what currently happens to a message that the consumer did not process are enough.
- List of topics and subscriptions, their purpose, who acts as publisher and who as consumer.
- How retries and the dead-letter queue are configured: after what number of attempts a message goes to dead-letter and who triages its contents.
- Roles and access rights: which accounts publish and read, where permissions are granted more broadly than required.
- What is already collected in monitoring: delivery latency, volume of the untriaged queue, alerts, and who receives them.
- Known failure points: load spikes, duplicates, message loss on consumer restart, manual incident triage.
- Requirements for the order of work: what must operate without downtime and which changes are acceptable during working hours.
Let's discuss the task in a review format: we will go through the exchange scheme, mark bottlenecks and failure points, and fix what we do first.
Describe your situation — we will come back with an estimate and a work plan, with stages, areas of responsibility, and a way to verify the result.