Change Streams MongoDB: where the change stream breaks
The typical pattern looks like this: services read not the primary database but a copy that is updated on a schedule — once an hour or overnight.
Between a real change and its appearance in the copy there is a window, and everything that happens inside it is simply invisible to services. While the load is low, this goes unnoticed; under a stream of orders, inventory, or statuses, discrepancies begin to accumulate.
The cost is stale data in services: a customer sees a product in stock that is already gone, the request status lags by several steps, reporting calculates from yesterday's snapshot.
Such database synchronization problems are insidious because they do not always reproduce and not for a single user, so they are blamed on the cache or on a "floating" defect. Then it is the business logic that breaks, not just a single screen.
The most expensive part begins during incident investigation. You need to understand which exact change did not arrive and at what moment — but there is no log, only two snapshots that do not match.
Then comes manual data reconciliation: exports, key-based comparison, eyeballing discrepancies. This takes days of engineers' work, and a repeat failure is still not excluded.
Change Streams MongoDB close this gap differently: the database itself gives the subscriber a continuous MongoDB change stream — inserts, updates, deletes — as they occur, without polling on a schedule.
The stream has a resume point, so after a restart the service continues from where it stopped, not from the overnight snapshot. Below we examine where such a stream breaks in practice and what we do to keep it from losing events.
What a change subscription provides: measurable effect
A change subscription to MongoDB turns the database from a source that has to be polled into an event source. Below is what changes in operations when real-time data synchronization and event-driven architecture are built on this mechanism.
Reaction latency — seconds instead of minutes
Previously synchronization ran by scheduled polling: minutes passed between a write and the appearance of data in the read model, search, or notification.
A MongoDB change subscription delivers the event immediately after the write is committed, so lag is measured in fractions of a second, and the polling interval no longer determines the system's accuracy.
Load on the primary database drops by an order of magnitude
Polling “has anything changed?” hits the database constantly, and each new consumer adds its own queries.
A subscription reads the operations log once and distributes events to subscribers: the number of requests to the primary database stops growing with the number of services, and processing does not compete with client traffic at peaks.
Manual desynchronization investigations go away
Fault-tolerant event delivery is built on processing acknowledgement and a saved resume position: after a restart the stream resumes from the same place, and duplicates are suppressed by event ID.
The share of “data diverged” incidents that used to be investigated manually after a complaint drops to zero.
Desynchronization is visible before the customer sees it
Each stream has a lag metric: how many events are not yet processed and how long ago the last one arrived.
The failure class “the export silently did not arrive” moves from after-the-fact detection to observability — an alert arrives when lag crosses the threshold, not when a user notices a discrepancy.
Change subscription modes in MongoDB: comparison of approaches
In MongoDB, change delivery is handled by one server-side mechanism — a change stream based on the operations log.
The subscription modes, however, are different, and they are not interchangeable: the granularity of the stream, the place of filtering, and how many independent event recipients can be held on one cursor differ.
The choice comes down to three questions: how narrow a stream the consumer needs, how many recipients there are, and what latency is acceptable. Below is a comparison of synchronization approaches that are actually chosen in practice.
| Option | When it fits | Limitations |
|---|---|---|
| Collection subscription with server-side stream filtering | One or two consumers, need events of specific types | The filter must be rebuilt when logic changes; under load, events arrive in batches |
| Database or entire cluster subscription | Data is spread across many collections, and the composition changes | The stream is wider, filtering falls to the consumer, and read volume grows |
| Collection polling by a marker field (modification time or version counter) | No access to the operations log, need a predictable mechanism | Latency equals the polling interval; gaps are possible when the marker coincides |
| Writing to an intermediate log in the same transaction as the data | Need guaranteed delivery and multiple recipients | Additional write and a dedicated sender service |
| Database-side handlers | Reaction without a separate consumer service | Logic lives next to the data, versioning and debugging are more difficult, rollback is limited |
They are often combined: a narrow subscription plus an intermediate log if there are multiple recipients.
Each consumer gets its own cursor and its own saved position: a slow handler then does not drag the others along, and after a failure the stream continues from the last confirmed event.
Implementation stages: from audit to handover to operations
The order of work here is not a formality: a change subscription relies on the replica set operations log, and if it is configured before the audit, the stream either loses events when the log overflows or silently lags during write peaks.
Therefore, implementing change streams in MongoDB goes from measurements to load testing and only then to operations.
- Database audit. We review the replica set topology, write rate, operations log size and growth rate, and load profile: point inserts, bulk updates, rare deletes. We separately record who else reads the same collections — those processes share resources with the stream.
- Event stream schema. We determine the collections and operation types included in the subscription, filters by conditions and fields on the database side. We fix the output event format and the destination it goes to: queue, handler, or storage.
- Change subscription configuration. We set the full document mode, resume token storage, and restart-from-checkpoint rules. We make the handler idempotent — a repeated event must not break state.
- Event delivery verification. We run an end-to-end scenario: change in the database → event → handler → result. We observe behavior on connection loss, service restart, and an intentional duplicate.
- Stream load testing. We measure delivery latency and lag at peak write rate, and check what happens when the log overflows. Based on the results, we adjust batch size, buffers, and the number of consumers.
- Handover to operations. We provide the stream schema, configuration, metrics and alert thresholds, and the procedure for lag. The customer's team sees the entire stream and knows where to look.
What your team keeps after implementation
We do not leave the implementation “working but unclear”: together with the configured change stream, your team receives a kit that allows it to maintain the system independently, and a new engineer can get up to speed in a couple of days.
- Change stream configurations — files with connection parameters, fetch batch size, document completeness level, and resume strategy after connection loss. Each value is commented: what will change in behavior if it is increased or decreased.
- Data flow schemas — where a change comes from, what processing stages it goes through, where the result is written, and where protection against redelivery of the same event is in place.
- Access policies and roles — separate accounts for reading the stream and for service operations, minimally required privileges, and key rotation procedure.
- Change stream documentation — which collections and fields are tracked, how the operations log is read, which events are considered service events and discarded.
- Operations procedure — how to check processing lag against the current moment, what to look at during a break, and which thresholds require an on-call intervention.
- Change rollback procedure — how to return the previous configuration and stop consumers without losing already read events.
- Set of metrics and alerts — delivery latency, share of processing errors, number of reconnections.
Each artifact is tied to a specific operation rather than written “for a checkbox”: using the procedure, the on-call engineer understands what to do during a break, and from the schemas it is clear where a new consumer can be added without revising the entire architecture.
Case study: order synchronization without manual reconciliation
The client is an order service: state changes in the primary database, while the storefront, warehouse, and billing picked up changes by periodic polling every fifteen minutes.
At peaks the polling could not keep up, lag reached forty minutes, and on connection loss changes were lost: the next request saw the next state and “swallowed” intermediate status transitions.
The share of discrepancies for active orders stayed within 3–5%, and every week support manually investigated several dozen such cases.
What we changed
The change stream was moved to Change Streams MongoDB: each consumer is subscribed to its own collection with a filter by operation type and sees events in the order they occur.
The resume token is written after processing each event, so after a break the subscriber continues exactly from the stopping point — without gaps and without rereading the entire history. Processing was made idempotent by event ID so that redelivery does not double charges.
What the result was
Synchronization lag at peak is seconds instead of tens of minutes, the share of discrepancies fell below 0.1%, and manual investigations went from dozens per week to isolated cases.
Resilience was tested with a controlled connection break: the subscriber caught up on what it missed by itself, with no on-call intervention required.
The warehouse service and billing received a subscription to the same events — synchronization of several services now comes from one stream instead of three independent polls.
What to do if events are lost during a connection break
Events are lost not at the moment of connection loss with the database, but at the moment of reconnection.
If the ID of the last processed event lived in the process memory, then a container restart or application restart means starting from the current moment — everything that entered the database during downtime does not exist for the handlers.
In MongoDB this failure looks like “the stream is alive, there are no errors, but some changes were not applied.”
The stream resume point is the event ID that the server returns with each record and can accept back in the resumeAfter and startAfter options. The algorithm is simple: we record the ID only after successful processing of the event, in reliable storage next to the business data.
The order “first wrote the point, then processed” causes loss if it crashes between the two steps; the reverse order causes redelivery, but never a gap.
Restoring the change stream after a break works through the operations log: the server replays accumulated changes starting from the passed point.
The problem begins when the pause exceeds the log depth — then the point is considered invalid, the stream will not come up, and a cold start with a full reconciliation of collection state against events is needed.
That is why processing lag and log depth are kept under monitoring with an alert, rather than discovered after the fact.
Redelivery is standard behavior; the guarantee here is “at least once.” Handlers are made idempotent: an event key in a deduplication table or an upsert by natural key.
During quiet periods events do not arrive, and the resume point does not move — it is advanced by a service batch marker, otherwise after a long lull the history will again end up outside the log and the stream will have to be recreated.
Practice of our engineers: the resume point is stored outside the process, handlers are idempotent, and alerts are configured for lag and log retention period. Describe how your stream is currently set up — we will come back with an analysis of weak points and a correction plan.
Technical concerns: limitations and event ordering
The MongoDB change stream removes most questions during a demo and almost all of them during pre-launch review. The answers below determine whether the subscription will survive node failure, consumer lag, and change log growth.
Will an event be lost if the subscriber crashes?
Delivery is at least once: acknowledgement is sent after processing, so on failure the event will arrive again. We make consumers idempotent — the operation key and a check of the applied change suppress duplicates.
Is event ordering guaranteed?
Order is maintained within a single routing key, not the entire database. Where strict sequence by object is important, we bind subscribers to the same key — otherwise two updates to the same document will arrive in a different order.
What happens if the subscriber falls behind by hours?
The read position is stored by a resume token, but the availability window is limited by the size of the change log. As soon as the position falls outside the boundary, the stream cannot be restored — reinitialization is required. We keep the window with a margin for maximum downtime.
How much load does this put on the database?
The stream reads the change log, not collections, and mainly loads the read node. We move subscribers to a replica, set a separate connection and limits — production queries do not compete with event processing.
What limitations do the streams themselves have?
Replication mode is required — a subscription cannot be set up on a single node. Some collection-level operations are not reflected fully, and field filters are sensitive to collation. We cover these areas with regular reconciliation.
Let's discuss the task: what to send for an implementation assessment
To come back with an implementation assessment rather than general words, we need a snapshot of the system: how the data is structured, who reads the MongoDB change stream, and what is critical for you in its behavior. Below are inputs that cover almost all unknowns at the start.
- Collection schema and write pattern: which collections enter the stream, the share of inserts, updates, and deletes, whether there are bulk edits and transactions.
- List of event consumers: services, queues, search index, cache, analytics storage — and who needs the full document versus only the key and operation type.
- Event ordering requirements: whether redelivery is acceptable and within what window, whether strict ordering within a document is needed, and what to do on a gap.
- Failure behavior: how long a consumer may be unavailable, whether buffering is needed, and from what position the stream resumes after a failure.
- Operations log horizon and planned maintenance procedure: how far back the stream can be rewound and what happens during a prolonged stop.
- Access and target indicators: roles, network restrictions, acceptable delivery latency, and peak stream volume.
Send what you have: collection schemas, a list of consumers, current procedures. We will discuss the task — and come back with an implementation assessment and work plan: stages, areas of responsibility, and checkpoints.