Kafka Topics and Partitions Configuration: A Tuning Guide

Kafka Topics and Partitions Configuration: A Tuning Guide

Imagine: you go live in production, and consumer lag grows, partitions are skewed, one broker is overloaded. A typical scenario with incorrect Kafka topic configuration. We'll walk through how to design a topic schema to avoid these issues.

Partition count and replication factor are two parameters that you cannot easily change after creating a topic. Reducing the number of partitions is impossible without fully recreating the topic. So correct initial configuration is vital. We guarantee that with proper design, you won't face hot spots or unnecessary delays.

Why Correct Partition Configuration Matters

Each partition is a unit of parallelism. One consumer in a group processes one partition. If a topic has 6 partitions, at most 6 consumers in the group can read in parallel. Extra consumers sit idle. Plus, writes within a partition are strictly ordered. Global ordering across a topic is not guaranteed—only within a partition. This is critical for events that must be processed sequentially (e.g., all actions of a single user).

Example: topic user-events with 6 partitions. Events from a single user (user:101) could land in different partitions (0 and 1), breaking processing order. A message key solves this: hash(user_id) % num_partitions always yields the same partition.

How to Create and Configure Topics

Via kafka-topics.sh

# Basic topic for user events kafka-topics.sh --bootstrap-server kafka-1:9092 \ --create \ --topic user-events \ --partitions 12 \ --replication-factor 3 \ --config retention.ms=604800000 \ --config retention.bytes=10737418210 \ --config compression.type=lz4 \ --config min.insync.replicas=2 \ --config message.max.bytes=1048576 # Compact topic — for storing the latest state by key kafka-topics.sh --bootstrap-server kafka-1:9092 \ --create \ --topic user-profiles \ --partitions 24 \ --replication-factor 3 \ --config cleanup.policy=compact \ --config min.cleanable.dirty.ratio=0.1 \ --config segment.ms=3600000 \ --config delete.retention.ms=86400000 # High-priority queue with short retention kafka-topics.sh --bootstrap-server kafka-1:9092 \ --create \ --topic order-processing-priority \ --partitions 6 \ --replication-factor 3 \ --config retention.ms=3600000 \ --config max.message.bytes=102400 

Programmatic management via Admin API (Java/Kotlin) Creating topics programmatically is ideal for applications that create topics dynamically:

Properties props = new Properties(); props.put(AdminClientConfig.BOOTSTRAP_SERVERS_CONFIG, "kafka-1:9092,kafka-2:9092,kafka-3:9092"); props.put(AdminClientConfig.REQUEST_TIMEOUT_MS_CONFIG, 5000); props.put(AdminClientConfig.DEFAULT_API_TIMEOUT_MS_CONFIG, 10000); try (AdminClient admin = AdminClient.create(props)) { NewTopic userEvents = new NewTopic("user-events", 12, (short) 3); userEvents.configs(Map.of( "retention.ms", "604800000", "compression.type", "lz4", "min.insync.replicas", "2" )); NewTopic deadLetter = new NewTopic("user-events-dlq", 3, (short) 3); deadLetter.configs(Map.of( "retention.ms", "2592000000", // 30 days "retention.bytes", "-1" )); CreateTopicsResult result = admin.createTopics(List.of(userEvents, deadLetter)); result.all().get(30, TimeUnit.SECONDS); } 

Modifying an existing topic's configuration

# Increase retention kafka-configs.sh --bootstrap-server kafka-1:9092 \ --alter \ --entity-type topics \ --entity-name user-events \ --add-config retention.ms=1209600000 # Add partitions (only increase!) kafka-topics.sh --bootstrap-server kafka-1:9092 \ --alter \ --topic user-events \ --partitions 24 # Caution: adding partitions breaks ordering for keyed messages. # Existing keys will go to the same partitions (hash % 12), # new keys will be distributed across 24 partitions. # View topic configuration kafka-configs.sh --bootstrap-server kafka-1:9092 \ --describe \ --entity-type topics \ --entity-name user-events 

Leader Management and Skew Mitigation

Uneven distribution of leaders among brokers leads to hot spots:

# Check leader distribution kafka-topics.sh --bootstrap-server kafka-1:9092 \ --describe --topic user-events # Preferred replicas — rebalance leaders kafka-leader-election.sh --bootstrap-server kafka-1:9092 \ --election-type preferred \ --all-topic-partitions # Or for a specific topic via JSON cat > election.json << 'EOF' { "partitions": [ {"topic": "user-events", "partition": 0}, {"topic": "user-events", "partition": 1} ] } EOF kafka-leader-election.sh --bootstrap-server kafka-1:9092 \ --election-type preferred \ --path-to-json-file election.json 

If consumer lag varies significantly across partitions, check the keys: a poor hash function causes skew. Solution: increase the number of partitions and use UniformStickyPartitioner (available from Kafka 2.4+). Alternative: switch to RoundRobinPartitioner for logging. Monitor partitions using kafka-consumer-groups.sh:

# Consumer lag — group lag kafka-consumer-groups.sh --bootstrap-server kafka-1:9092 \ --describe --group my-consumer-group # Total lag > 10,000 for critical topics — trigger for alert 

Typical Configurations by Data Type

Comparison: Kafka vs RabbitMQ for Data Streams

Apache Kafka is better than RabbitMQ by 3–5 times in throughput under high load (hundreds of MB/s). RabbitMQ wins in flexible routing (exchanges) and support for queues with different priorities. For event sourcing and CDC, Kafka is the de facto standard; for microservices with complex routing, RabbitMQ may be more convenient.

Our Process and Timeline

With over 10 years of experience and 40+ successful Kafka deployments, our clients typically save $30,000 per year by optimizing Kafka topics. Our approach is:

• Requirements analysis: evaluate throughput, consumer count, ordering needs, retention. Design topic schema. Duration: 1 day.
• Topic creation: configure partitions, replication, compaction. Set up ACLs if authentication is needed. Duration: 1–2 days.
• Monitoring: configure consumer lag alerts, document schema for the team. Duration: 1 day.

We offer turnkey configuration: from design to documentation. We can assess your project in 1 day — contact us.

What's Included?

• Architectural documentation: topic schema, keys, retention.
• Broker and topic configuration.
• Monitoring and alert setup.
• Team training (2–4 hours).
• One month of post-launch support.

Timeline

Proper partition configuration reduces latency by 30% compared to suboptimal setups (based on our data). Get a consultation for your project — reach out to us.

Learn more about Apache Kafka at Wikipedia.