Apache Kafka Cluster Setup for Web Applications

Setting up an Apache Kafka Cluster for a Web Application

Imagine your web application processes thousands of orders per minute. Every event — sending an email, updating a search index, syncing data — happens synchronously. Servers choke, latency grows, and failures cause data loss. Apache Kafka solves these problems, but incorrect cluster configuration (e.g., single broker with replication factor = 1) leads to disaster. One of our clients — an e-commerce platform handling 500,000 events per second — migrated to Kafka after RabbitMQ failed under peak load.

Kafka is a distributed log with ordering and replication guarantees. It decouples services, provides an audit log, and enables real-time analytics. A production cluster requires careful setup: mode selection, broker configuration, security, and monitoring. Our team has deployed over 30 clusters for highload projects — from fintech to e-commerce. Proper configuration can reduce infrastructure costs by up to 40% compared to alternatives, with payback within six months.

How to Set Up an Apache Kafka Cluster: KRaft or ZooKeeper?

For new projects, use only KRaft (no ZooKeeper). Since KRaft became production-ready in version 3.3, it simplifies architecture, reduces failure points, and speeds up recovery. ZooKeeper remains for legacy setups, but we do not recommend starting with it.

Source: Kafka official documentation

Installation and Configuration on Ubuntu

Install the latest stable version, set up the systemd service, and initialize storage:

apt install -y openjdk-21-jdk-headless
KAFKA_VERSION=3.7.0
SCALA_VERSION=2.13
wget https://downloads.apache.org/kafka/${KAFKA_VERSION}/kafka_${SCALA_VERSION}-${KAFKA_VERSION}.tgz
tar -xzf kafka_${SCALA_VERSION}-${KAFKA_VERSION}.tgz -C /opt/
ln -s /opt/kafka_${SCALA_VERSION}-${KAFKA_VERSION} /opt/kafka
useradd -r -s /bin/false kafka
chown -R kafka:kafka /opt/kafka
mkdir -p /var/log/kafka /data/kafka
chown kafka:kafka /var/log/kafka /data/kafka
cat > /etc/systemd/system/kafka.service << 'EOF'
[Unit]
Description=Apache Kafka
After=network.target

[Service]
Type=simple
User=kafka
Environment="KAFKA_HEAP_OPTS=-Xmx4g -Xms4g"
Environment="KAFKA_JVM_PERFORMANCE_OPTS=-server -XX:+UseG1GC -XX:MaxGCPauseMillis=20 -XX:InitiatingHeapOccupancyPercent=35 -XX:+ExplicitGCInvokesConcurrent -Djava.awt.headless=true"
ExecStart=/opt/kafka/bin/kafka-server-start.sh /opt/kafka/config/kraft/server.properties
ExecStop=/opt/kafka/bin/kafka-server-stop.sh
Restart=on-failure
RestartSec=5
LimitNOFILE=65536

[Install]
WantedBy=multi-user.target
EOF
CLUSTER_UUID=$(kafka-storage.sh random-uuid)
kafka-storage.sh format -t $CLUSTER_UUID -c /opt/kafka/config/kraft/server.properties

To tailor the configuration to your project, contact us.

KRaft configuration on each node (example for node 1; for nodes 2 and 3 change node.id and advertised.listeners):

node.id=1
process.roles=broker,controller
controller.quorum.voters=1@kafka-1:9093,2@kafka-2:9093,3@kafka-3:9093
listeners=PLAINTEXT://0.0.0.0:9092,CONTROLLER://0.0.0.0:9093
advertised.listeners=PLAINTEXT://kafka-1.internal:9092
inter.broker.listener.name=PLAINTEXT
controller.listener.names=CONTROLLER
listener.security.protocol.map=PLAINTEXT:PLAINTEXT,CONTROLLER:PLAINTEXT
log.dirs=/data/kafka
num.recovery.threads.per.data.dir=4
num.io.threads=16
num.network.threads=8
socket.send.buffer.bytes=1048576
socket.receive.buffer.bytes=1048576
socket.request.max.bytes=104857600
default.replication.factor=3
min.insync.replicas=2
num.partitions=6
offsets.topic.replication.factor=3
transaction.state.log.replication.factor=3
transaction.state.log.min.isr=2
log.retention.hours=168
log.segment.bytes=1073741824
log.retention.check.interval.ms=300000
compression.type=lz4

Configuring TLS Between Brokers

Without TLS, traffic is transmitted in plain text. For protection, use your own CA and certificates for each broker. Detailed instructions for generating certificates will be provided as part of the project.

Add to server.properties on each broker:

listeners=PLAINTEXT://0.0.0.0:9092,SSL://0.0.0.0:9094,CONTROLLER://0.0.0.0:9093
ssl.keystore.location=/etc/kafka/ssl/kafka-1.keystore.jks
ssl.keystore.password=changeit
ssl.key.password=changeit
ssl.truststore.location=/etc/kafka/ssl/kafka.truststore.jks
ssl.truststore.password=changeit
ssl.client.auth=required
ssl.enabled.protocols=TLSv1.3,TLSv1.2

Monitoring: Key Metrics and Alerts

Track under-replicated partitions, controller activity, p99 producer/consumer latency, consumer lag. Use JMX Exporter + Prometheus for collection, Grafana for visualization. Example JMX Exporter configuration:

---
startDelaySeconds: 0
hostPort: 127.0.0.1:9999
lowercaseOutputName: true
rules:
  - pattern: kafka.server<type=BrokerTopicMetrics, name=MessagesInPerSec><>OneMinuteRate
    name: kafka_server_broker_topic_messages_in_per_sec
  - pattern: kafka.server<type=ReplicaManager, name=UnderReplicatedPartitions><>Value
    name: kafka_server_under_replicated_partitions
  - pattern: kafka.controller<type=KafkaController, name=ActiveControllerCount><>Value
    name: kafka_controller_active_count
  - pattern: kafka.network<type=RequestMetrics, name=TotalTimeMs, request=Produce><>99thPercentile
    name: kafka_network_produce_total_time_ms_p99
---

Key alerts: kafka_server_under_replicated_partitions > 0, kafka_controller_active_count != 1, consumer lag above threshold. Contact us to implement monitoring and get ready-made dashboards for your load.

How to Ensure Fault Tolerance?

1. Create a test topic with replication factor 3.
2. Shut down one broker and verify that producers do not lose data (with acks=all and min.insync.replicas=2).
3. Bring the broker back and ensure replicas synchronize.
4. Check consumer lag — it should return to zero.
5. Repeat for each broker.

kafka-topics.sh --bootstrap-server kafka-1:9092 --create --topic test-topic --partitions 6 --replication-factor 3
kafka-producer-perf-test.sh --topic test-topic --num-records 1000000 --record-size 1024 --throughput -1 --producer-props bootstrap.servers=kafka-1:9092,kafka-2:9092,kafka-3:9092 acks=all compression.type=lz4
kafka-consumer-perf-test.sh --bootstrap-server kafka-1:9092 --topic test-topic --messages 1000000 --group perf-test-group

We guarantee SLA 99.99% for your cluster. Order deployment — contact us for a consultation. Get a preliminary cost and timeline estimate.

Process and Timelines

Total: 5-7 working days for a basic cluster. Cost is calculated individually.

What's Included

• Cluster architecture design (roles, partitions, replication)
• Installation and configuration in KRaft mode
• TLS configuration between brokers and clients
• Monitoring implementation (JMX Exporter + Prometheus + Grafana)
• Creation of production topics with optimal parameters
• Load testing and fault tolerance verification
• Documentation (cluster diagram, instructions, runbook)
• Team training (basic administration, CLI usage)
• Technical support for 2 weeks after launch

Results and Guarantees

Over 30 deployed clusters with loads up to 1 million messages per second. Average infrastructure cost reduction — 35%. Payback time — up to 6 months. Contact our engineers — we will help configure a cluster for your load.