Configuring Apache Kafka: where the cluster loses data
Message loss in a cluster is rarely related to the platform itself: the stream breaks at configuration junctions, and it is noticed only through reconciliations and reports.
The most frequent cause is partition imbalance: the partitioning key is chosen such that one or two partitions take almost the entire stream, their brokers hit disk limits, the rest sit idle, and consumer lag grows without any failure.
Next — replication without loss is only on paper. A replication factor greater than one guarantees nothing by itself if write acknowledgment is configured from the leader and does not wait for synchronous replicas.
When a node fails, the latest records disappear, and consumer offsets have already advanced — there is nothing to restore from, because those messages were not in the replicas.
A separate class is capacity. Page cache memory, segment size, retention period and volume are interconnected: with incorrect limits, a node goes read-only or deletes segments before lagging groups have read them. The application does not receive an error — it simply undercounts events a day later.
Finally, observability. Without lag metrics for each consumer group, replica state, and alerts on sustained lag growth, an incident is discovered from user complaints.
Our Apache Kafka setup begins with an audit of the partition scheme, write acknowledgment parameters, and retention policies, and ends with cluster expansion and node failure occurring without manual rebalancing and without event loss.
Performance and fault tolerance under peak loads
Peak load tests not average throughput but the tail: with a multiple increase in event stream, the bottleneck shifts to replication, partition reassignment, and leader election.
A well-tuned Apache Kafka configuration turns this into normal operation — the cluster handles growth by factors, and publish latency grows linearly, not in steps.
KRaft mode with a controller quorum removes external coordination: metadata lives inside the cluster, and electing a new partition leader takes seconds instead of minutes.
Recovery time after a node failure becomes predictable — with a replication factor of 3 and brokers spread across availability zones, the cluster accepts writes as long as the majority of replicas are online.
For critical topics, acks=all and min.insync.replicas prevent a write from being acknowledged before several replicas accept it — the share of lost messages drops to zero even if a broker fails at the moment of writing.
An idempotent producer and transactional writes remove duplicates that arise from resending after a network failure: there are single-digit repeats per millions of events instead of percentages.
Headroom is visible in three metrics: consumer lag, the number of partitions outside the in-sync replica set, and group rebalance duration. Thresholds for them are set in advance, so peak resilience is tested in drills, not during peak reporting load.
Operating modes: single node, self-managed cluster, managed service
The operating mode determines who is responsible for controller availability, replication, and updates. In KRaft, metadata is held by a controller quorum: on a single node there is none at all, so a disk failure stops both writes and reads.
Therefore, topology choice depends on fault tolerance and control requirements, not on topic volume.
The decisive parameters are replication factor, minimum number of in-sync replicas, number of broker placement zones, and data isolation requirements.
If regulations prohibit moving data outside your perimeter, the managed option is out; if there is no on-call team, a self-managed cluster becomes a risk.
| Option | When it fits | Limitations |
|---|---|---|
| Single node, broker and controller together | Staging, development, stream within the capabilities of one broker | No replication: node failure stops writes and reads, updates only with downtime |
| Self-managed cluster, three or more brokers in different zones | Data must remain in your perimeter, full control over parameters and versions is needed | Controller quorum, replication, certificates, and Schema Registry are on your side |
| Managed service from a provider | Fault tolerance is needed, but you have no support team | Some broker parameters are closed, version is set by the provider, rights and security settings are limited |
We fix the topology before configuring TLS and Schema Registry: listener names, client addressing scheme, and certificate rotation order depend on it.
We move from a single node to a cluster by recreating topics with the required replication factor and migrating consumers with offset commits — without stream loss.
How implementation proceeds: from audit to handover to operations
The Kafka implementation stages are structured so that each ends with a verifiable artifact — a configuration, a scheme, or a load test report. Production traffic is not cut over "in one night" but after the new perimeter has proven resilience on a realistic profile.
- Cluster audit. We capture the actual state: number of nodes, partition layout, replication factor, lagging replicas, retention periods, and current rights. The output is a map of bottlenecks and a list of parameters to change.
- Partition and topic design. We design a model for the stream profile: partitioning key, partition headroom for growth, separate topics for data with different ordering and acceptable loss requirements.
- Node and access configuration. We set up the perimeter: traffic encryption, authentication, topic- and consumer-group-level rights, quotas, message size limits. We choose the operating mode to remove an extra coordination layer.
- Schema registry and compatibility. We fix format contracts and the compatibility rule so that a new producer version does not break existing consumers. Mismatch is filtered at the entrance, not in analytics.
- Load testing. We run a profile close to production: peak stream, latency, node failure, replica recovery. Thresholds at which the cluster is healthy are embedded into monitoring and alerts.
- Handover to operations. We hand over regulations, schedules, and response rules, train the on-call team, and go through a training switchover together. Further support follows documented procedures.
Configurations, schemes, and regulations that remain with you
A working cluster by itself does not solve the problem: if after six months no one remembers why such replication parameters were chosen and how rights were issued to services, any change turns into excavation.
Therefore, we hand over the perimeter together with a full set of artifacts — from them, a new shift of engineers gets up to speed without correspondence with the authors.
- Configuration files for brokers, controllers, and the schema registry — with comments on what each parameter is responsible for and what will happen if it is changed.
- Cluster scheme: node topology, roles, failure zones, partition and replica distribution, failure points, and load headroom.
- Access policies: list of service accounts, their rights to topics and groups, key rotation rules, and the procedure for issuing new access credentials.
- Backup regulations: what is backed up (configurations, schemas, group state), how often, where it is stored, and how recovery is verified.
- Upgrade regulations: sequence of actions per node, readiness criteria for moving to the next, mandatory checkpoints, and the procedure for rolling back to the previous state.
- Incident troubleshooting instructions: typical failures, what to look at first, which metrics and logs to collect, and when to escalate.
- Team instructions: how to create a topic and register a schema, how to connect a new consumer, which checks to run before deployment.
Documentation lives in your repository and is updated together with the configuration, not separately from it. Describe your task — we will come back with a scope estimate and work plan, including the composition of the artifacts to be handed over.
Case study: failures once a day instead of a stable stream
Initial perimeter
One of our customers is a service with an event stream from dozens of sources: payments, order statuses, device telemetry. A three-broker Apache Kafka cluster kept metadata in a separate coordinator, topics were created manually, and message retention was not limited at all.
Disks filled up, a broker dropped out, consumer groups went into rebalance, and lag during peak hours grew to several hours.
What was rebuilt
Metadata was migrated to the built-in KRaft consensus — the separate coordinator was removed along with another point of failure.
For topics, replication factor 3 and a quorum of 2 in-sync replicas were fixed: a write is considered acknowledged only when at least two copies have accepted it.
The schema registry was deployed in clustered mode with compatibility checking on every change, TLS and certificate rotation were enabled for all clients without exception, and message retention was set separately per topic.
Metrics before and after
Before: a failure once a day, recovery 30–40 minutes, peak lag up to three hours, loss of acknowledged events during leader switchover.
After: failures requiring manual intervention — zero over several months of operation; lag during peak hours stays within tens of seconds, recovery from node failure takes a few minutes, and there is no loss of acknowledged events.
What to do if a Kafka cluster is already deployed
A running Apache Kafka cluster almost never has to be rebuilt from scratch — in most cases, an audit and a phased plan are enough.
First, we capture the state: number of brokers, topic replication factor, min.insync.replicas value, accumulated consumer group offsets, and peak write hours. These numbers determine what can be changed on a live system and what requires a separate window.
Migration to the built-in quorum instead of a separate coordinator goes in three steps: metadata conversion, adding controller nodes, and switching brokers to a mode where they see both state sources.
Each step is reversible until the final shutdown of the coordinator, and restart is done one broker at a time — replication maintains availability, and consumers do not notice downtime.
Upgrading on a live system follows an order: first the inter-broker protocol is raised, then the brokers themselves are upgraded. A new broker understands requests from an old client, so applications do not need to be rebuilt.
We check the schema registry separately: the chosen compatibility mode determines the order in which new versions are registered, and violating it is a common cause of producer failures immediately after an upgrade.
On a running system, adding brokers, partition reassignment, storage tuning, certificate rotation with a period when both are considered trusted, and access rights edits are safe.
Changing the partitioning key and reducing the replication factor without disk headroom are dangerous: message layout changes and processing order breaks. We assemble the migration plan in stages with explicit rollback points and a criterion for stopping and going back.
Common technical questions about cluster operations
Cluster operations run into the same questions: how to change the message format without stopping consumers, how to separate access between teams and external clients, what happens when a node fails, and in what order scheduled upgrades proceed. Below are short answers to what is most often clarified before work starts.
How to change the message schema without stopping consumers?
The schema is stored in the schema registry with a compatibility rule enabled: new fields are added only with default values.
Rollout goes in the order "consumers first, then producers", and the old schema version remains available — rollback does not require rebuilding messages.
How to restrict access to the cluster for external clients?
Each service or client gets its own account with rights only to the required topics: read or write, no broader. Traffic goes over an encrypted channel with client certificate verification, and speed is limited by quotas so that one consumer does not occupy the entire cluster.
What happens when a node fails?
Partitions are replicated to several nodes, so when a failure occurs, the leader automatically moves to a replica, and writing continues. We regularly run node failure in a test environment and verify that no consumer has lost messages.
How do scheduled upgrades and cluster support proceed?
We upgrade one node at a time — first replicas, then primary ones, and the service keeps running. Support includes monitoring replica and consumer lag, disk space control, incident troubleshooting, and agreed maintenance windows.
Let's discuss the task: we will come back with an estimate and work plan
To come back with an estimate and work plan rather than general statements, we need initial data from the customer. Below is a set that covers 90% of questions at the start: it shows the scope of work, risks, and sequence of actions.
- Perimeter and topology. How many brokers, how they are spread across availability zones, who acts as controllers, whether there is a separate perimeter for testing.
- Symptoms. What exactly hurts: replica lag, frequent consumer rebalances, latency growth at peak, schema update errors, manual configuration edits.
- Stream volumes. Number of topics and partitions, daily message volume, retention period and size, load profile — steady or with peaks.
- Data schemas. Whether there is a schema registry, what message formats are used, how version compatibility is currently checked when new services are released.
- Security and access. How clients authenticate, whether TLS is enabled, who owns certificates and how they are renewed, whether topic-level rights separation is needed.
- Maintenance window constraints. Acceptable downtime, whether brokers can be restarted one at a time, whether there is a period when load is minimal.
- Infrastructure and observability. Where everything is hosted, what storage system is used for logs, what is already being collected into monitoring and logs.
With this set, we will come back with an estimate and work plan: stages, order of changes, checkpoints, rollback scenario, and window requirements.
The next step is to discuss the task on a call: clarify details, agree on boundaries, and fix the plan with responsible persons and deadlines.