Setting Up NATS with JetStream and Clustering for Production

Why NATS loses messages under load

In the test environment, NATS holds the flow steady, so the first production spike looks like a mystery: some messages simply do not arrive.

The sender logs are silent, the metrics show a surge of failures, and gaps remain in the queue that are later restored by reconciling with an external source.

Message loss comes from three causes: a connection break when a node switches over, queue overflow at a slow consumer, and the absence of acknowledgements on the sender side.

Node switchover is the most common disconnect scenario. The leader goes away, the messaging cluster elects a new one, and during those fractions of a second the client writes to a dead socket and considers the data delivered.

If retries and acknowledgements are not configured, those messages are lost silently, with no error on the client. The same mechanism produces duplicates when retry is enabled but the handler is not idempotent: one event passes through twice and corrupts state.

The second source is delivery delays and queue overflow. When the consumer cannot keep up, the stream grows to the storage limit, and then either the policy for evicting old records kicks in or the sender gets a write rejection.

At peaks this looks like a ragged stream: some events arrive late, some disappear, and the queue recovers in a burst and overwhelms the handler with a delayed batch.

We will break down how this is addressed: acknowledgements and redelivery in JetStream streams, idempotent handlers, stream sizes and storage policies for the real load profile, client reconnection with buffer preservation, and node health checks.

What a fault-tolerant messaging cluster provides

A fault-tolerant NATS cluster is not “deployed three copies and forgot about it.” Its effects are verified before production launch: we shut down a node under load, look at the metrics, and decide whether we are ready for a real failure.

• A node failure does not stop the flow. A quorum of three nodes preserves writes when one machine drops out: publishing and subscribing go through live nodes, and clients reconnect on their own. Stream recovery time is seconds instead of service downtime.

• Delivery guarantee is set explicitly. Messages remain on disk until processing is acknowledged, so restarting the receiving service does not cause loss. Repeated processing is limited by deduplication and is visible in metrics rather than during incident analysis.

• Peak load does not turn into failures. If publishing is faster than processing, the difference settles in the storage buffer rather than becoming errors at the ingress — the service remains available during a spike.

• Updates proceed without manual restarts. Nodes are restarted one at a time, the quorum is preserved, and the on-call engineer does not manually bring up a failed process.

• Load boundaries are set in advance. Per-consumer limits and splitting streams by purpose prevent one service from taking resources from the others — degradation remains local.

We confirm each point in drills: we shut down a node under load, raise traffic above the calculated level, and update the cluster on a live stream. The result is that the service survives a machine failure and a peak without manual intervention.

NATS modes: stream, queue, and persistent storage

NATS modes solve different tasks, and the choice between them is determined by one question: what should happen to a message if the recipient is unavailable at the moment of sending.

The basic mode delivers an event to all subscribers of a subject at once and does not store it anywhere. A work queue distributes the stream between service instances — a message goes to exactly one worker. JetStream persistent storage writes the stream to disk and allows it to be re-read after a restart.

The nature of the data itself suggests the choice: what can tolerate loss goes over the subject without extra overhead; what requires reprocessing goes through JetStream with delivery acknowledgement.

Combining modes in one installation is normal: subject subscriptions and queues use the same transport, so switching between them is a change to subscriber settings, not a service rewrite.

Storage is enabled selectively, only for the required subjects, so you do not pay with disk space for all traffic. The age or size limit is set by the storage policy — the stream stops growing, while reprocessing after a failure remains possible.

How we deploy the cluster: stages of work

A NATS cluster is assembled not according to a standard instruction but for a specific messaging profile: one service cares about every request, while another publishes telemetry with acceptable loss.

Our order of work is fixed — it allows topology errors to be caught in the test environment, not when the queue has already stalled.

1. Messaging schema audit. We gather who publishes and who subscribes, which subjects are already in use, and where a message passes through a chain of services. We mark streams that are critical with respect to loss and delivery order.

2. Schema design. We determine the cluster composition, the number of replicas for streams, storage limits, and cleanup policies. The design immediately goes into a configuration in the repository, not into a “for memory” document.

3. Cluster setup. We deploy nodes, join them into a cluster, enable JetStream, set the quorum, access rights, and traffic encryption between nodes. We connect metrics and alerts for consumer lag.

4. Failure verification. We shut down nodes one at a time and in pairs: we look at leader re-elections, recovery time, availability of publishing and subscribing, and the absence of duplicates and losses. The output is measurements, not promises.

5. Handover to operations. We provide configurations, deployment and recovery instructions, a schema description, and on-call scenarios. Your engineers repeat the deployment on a clean environment without hints.

6. Support. We monitor cluster health, update versions on schedule, scale as load grows, and analyze incidents with your team.

Configurations, schemas, and regulations on delivery

We hand over artifacts not as a “memory archive” but as a working set: your team can read, modify, and deploy any of the listed entities without us.

Everything lives in your repository, is versioned, and is verified in the test environment — we run the configurations through a test cluster and make sure they reproduce from scratch.

• Node configurations — settings files for each node with explanations of key parameters: storage limits, message retention windows, timeouts, and behavior when quorum is lost.
• Cluster schema — route topology, failure zones, balancing points, and the procedure for adding and removing a node for maintenance.
• Subjects and subscriptions schema — a map of streams and queues: who publishes, who reads, which subscriber groups share the load, and where replay is enabled.
• Access control policies — by accounts and namespaces: who is allowed to publish, who can only read, who is denied access to the administrative scope, and how keys are issued and revoked.
• Operations regulation — scheduled maintenance, storage growth control, log rotation, and the procedure for updating nodes without stopping messaging.
• Recovery instructions — scenarios for losing a node or a zone: what is taken from reserve, in what sequence it is brought up, and how integrity is verified.
• Monitoring and alerts — a set of metrics and thresholds at which the on-call engineer receives a signal before the problem reaches business logic.
• Knowledge transfer — a walkthrough with your engineers for each artifact so your team can run the system independently.

The set covers the full cycle: deploy from scratch, change configuration without downtime, roll back an unsuccessful update, and hand the system over to another contractor without archaeology. Describe your task — we will come back with an estimate and a plan.

Case: NATS cluster instead of a custom queue

A company with billing and mass mailings kept its task queue on a custom solution: workers selected rows from a relational table by a status flag, and consistency relied on locks.

At peaks, the table hit row contention, jobs were duplicated, and some events disappeared without a trace — analyzing such an incident took an evening and required manual reconciliation with service logs.

What it was before

The share of lost messages during peak hours reached 0.3–0.5% of the stream, and payment confirmations fell into those percentages.

Recovering a node after a crash took about 40 minutes: workers did not redistribute jobs on their own, and the on-call engineer brought them up manually. Manual interventions added up to 8–12 per month — each one pulled an engineer away from planned tasks and shifted deadlines.

What we did

We deployed a NATS cluster with JetStream: streams with disk persistence and replication across several nodes, consumers with explicit delivery acknowledgement and redelivery on handler error.

We made handlers idempotent by event key so that retry would be safe, and replaced manual review of “stuck” jobs with automatic redelivery by timeout.

What the result was

On the critical payment confirmation stream, message loss is zero, including peak windows. A node returns to service in less than a minute: consumers reconnect to live replicas on their own, and manual bring-up is no longer needed.

On-call interventions have been reduced to scheduled updates, and stuck jobs are visible in stream metrics rather than discovered through complaints.

A separate effect is rollback predictability: queue state is stored on disk, so restarting the service does not mean losing the accumulated stream.

How a NATS cluster differs from a single broker

A single NATS node is one process and one storage: while it is alive, publishing and subscribing proceed with minimal latency, configuration is read from one file, and operations come down to restarting the service.

There is exactly one point of failure, and it coincides with the point where all messaging between services stops.

A cluster is three or more nodes connected by routes: they form a quorum, re-elect a leader, and keep copies of JetStream streams on different nodes. A failure of one node does not interrupt publishing: clients reconnect to the survivors, and the stream is restored from a copy.

But if two out of three nodes drop out, writes stop until the third returns — a deliberate price for data consistency.

In operations the difference is no smaller: a cluster needs consistent configurations, control over routes and replication, and distribution of nodes across availability zones.

In return, you get node-by-node updates without downtime and the ability to survive the failure of an entire zone. A single node is chosen where messaging can be interrupted during maintenance — and then a cluster would only complicate the environment.

Common technical doubts before starting

Before starting, the customer’s engineers check not us but compatibility with the environment already in operation. Below are the five most frequently asked questions, with brief answers that get to the point.

How do we know the cluster will handle our load?

We capture the profile of the real stream — subjects, message size, acknowledgement share, peaks — and run it on a test environment. We look at where latency grows and how storage behaves when one node fails.

We already have messaging deployed — is migration without downtime possible?

We migrate subject by subject, not all at once: for part of the time, messages go to both environments, and consumers switch over when ready. We disable the old route after reconciliation, so consumers experience no downtime.

Settings have diverged between environments — how is that fixed?

We move configuration into the repository and describe it declaratively: one source of truth for all environments, with differences only in environment parameters. Divergences are checked automatically before rollout.

What happens if the new configuration turns out to be unsuccessful?

Rollback takes minutes: the configuration is versioned, and accumulated messages remain in storage and are not lost on restart. We document the rollback scheme and verify it in the test environment in advance.

Let's discuss your messaging task

So that the estimate and plan are based on facts rather than assumptions, we need baseline data on your messaging environment.

The more accurately you describe the schema and load profile, the fewer assumptions we will build into the cluster and storage policy — and the more predictable the timeline will be.

• Messaging schema: who publishes, who reads and in which groups, how many streams and subscriptions there are, and what happens during a spike.
• Load profile: messages per second at peak and on average, typical and maximum message size.
• Durability requirements: acceptable loss during node failure, retention depth, and whether redelivery is needed when a consumer fails.
• Deployment environment: own server or cloud, how many zones, what communication channels exist between nodes, and how access is arranged.
• Current state: what is already deployed, where you observe bottlenecks, and whether there are logs, metrics, and descriptions of past failures.
• Organizational constraints: desired launch timeline, maintenance windows, and who on your side accepts the result.

Describe the task in this form — we will come back with an estimate and a work plan that includes the cluster composition, durability modes, and migration and rollback procedures. We will break the work format down into stages so that each step can be verified and accepted. We will discuss what we cover with our own resources and what requires changes on your side.