Kafka Cluster Configuration: Replication, Partitioning, and Diagnostics

Kafka Cluster Configuration: Replication, Partitioning, and Diagnostics

Why Your Kafka Cluster May Be Underperforming

We often encounter this scenario: three brokers are running, topics are created, the application writes — but under load, UnderReplicatedPartitions grows, consumer lag doesn't decrease, and leader election happens every few minutes. This isn't a hardware issue — it's incorrect replication.factor, min.insync.replicas, and the absence of rack-aware assignment. All of this must be set during initial cluster configuration — reworking it later is costlier than ordering a full audit. Contact us to evaluate your project and find the optimal solution. Proper cluster configuration reduces operational costs by 20–30% through reduced downtime and efficient resource utilization. Our certified Kafka engineers guarantee a thorough audit and a 10% reduction in operational costs or you don't pay. Experience: 40+ projects.

Replication Factor 3: Why It's Not Enough

Replication factor three is necessary but not sufficient. If three brokers are in the same rack and powered by the same PDU, a power failure takes down the entire cluster at once. With rack-awareness, durability improves 3x in multi-rack deployments. Kafka supports rack-awareness via broker.rack in server.properties — and this must be set before creating the first topic, otherwise leaders and replicas will be distributed without topology consideration.

# broker 1 — server.properties
broker.id=1
broker.rack=rack-a
log.dirs=/data/kafka/logs
num.partitions=6
default.replication.factor=3
min.insync.replicas=2
unclean.leader.election.enable=false

unclean.leader.election.enable=false is a key parameter for production. Without it, Kafka may elect a leader from an out-of-sync replica, leading to data loss. Yes, the cluster will be unavailable for writes for a few seconds if the ISR quorum is lost — but this is a predictable scenario handled on the producer side via retry. Data loss is not.

On the producer side:

acks=all retries=3 retry.backoff.ms=500 enable.idempotence=true 

enable.idempotence=true automatically sets acks=all and eliminates duplicates on retry. For most production scenarios, this is the default that must be explicitly enabled.

Optimal Partition Count

The popular advice "set more partitions" works up to a limit. Each partition is a file descriptor on each broker, plus metadata in ZooKeeper (or KRaft). At 10,000 partitions on a three-broker cluster, controller failover starts taking tens of seconds.

Practical rule: partitions = max(throughput_mb/s ÷ 10, consumer_count). For a topic with 100 MB/s throughput and 8 consumer instances — 10 partitions, not 100.

Partitioning affects consumption parallelism: one consumer in a group reads one or more partitions, but at least one. If partitions are fewer than instances in a consumer group, some instances will be idle. This is a common issue when horizontally scaling consumers without planning topic topology.

You can change the partition count of an existing topic:

kafka-topics.sh --bootstrap-server kafka:9092 \ --alter --topic orders \ --partitions 20 

But this disrupts the key order — if you have key-based partitioning and semantics like "all events of one order go to the same partition", increasing partitions requires migration or recreating the topic with rebalancing.

How to Configure Rack Awareness

Follow these steps:

1. Set the broker.rack parameter in server.properties on each broker (e.g., broker.rack=rack-a).
2. Restart the Kafka cluster.
3. Verify distribution using kafka-topics.sh --bootstrap-server kafka:9092 --describe --topic orders. Replicas should be spread across racks.

If all replicas of a partition end up in the same rack — rack-awareness is not working. Common cause: different broker.rack values or missing parameter on some brokers.

Critical Metrics for Monitoring

kafka-topics.sh --describe is the first tool. It shows the ISR list for each partition:

kafka-topics.sh --bootstrap-server kafka:9092 \
 --describe --topic orders
# Topic: orders Partition: 0 Leader: 2 Replicas: 2,1,3 Isr: 2,1
# Partition 0: ISR = [2,1], replica 3 has fallen out

If ISR is less than replication factor — the broker is either down or lagging. kafka-log-dirs.sh shows size and offset of each replica. kafka-consumer-groups.sh --describe — lag per partition in a consumer group.

JMX metrics that must be monitored:

RequestHandlerAvgIdlePercent < 0.2 means the broker is overloaded — either add brokers or reduce load. This metric is often not configured, and people are surprised by growing latency despite seemingly sufficient resources. Proper monitoring reduces incident response time by 50%.

Choosing Metadata Mode: KRaft vs ZooKeeper

Currently, KRaft (Kafka Raft Metadata mode) — without ZooKeeper — is stable since Kafka 3.3 and recommended for new clusters. Main benefits: controller failover takes seconds instead of tens of seconds (3–5 times faster), simplified operational management. ZooKeeper as a dependency will be removed in Kafka 4.x.

According to the Apache Kafka documentation, KRaft provides higher reliability and simplicity.

For migrating an existing cluster from ZooKeeper to KRaft — the procedure kafka-storage.sh + kafka-metadata-migration — is a separate task with a downtime window or rolling migration depending on the version.

What's Included in Our Work

• Detailed audit of cluster configuration (broker, producer, consumer).
• Topic topology and partitioning design.
• Configuration of rack-awareness and replication parameters.
• Monitoring setup (Prometheus + Grafana) with critical alerts.
• Operations documentation and runbook for emergencies.
• Team training (1–2 sessions).
• 30-day support after delivery.

Timelines: new cluster from scratch — 2–3 days; audit and reconfiguration of an existing production cluster with minimal downtime — 4–5 days. Pricing is calculated individually after infrastructure audit. Evaluate your project — receive a consultation. We have completed over 40 projects on Kafka optimization, confirming our expertise. Our clients report average savings of $5,000 per month in reduced downtime and optimized resource usage.