Elasticsearch Cluster Setup for Web Applications

Elasticsearch Cluster Setup for Web Applications

When moving from a single Elasticsearch instance to a cluster, you often run into non‑obvious problems: split‑brain, unbalanced shard distribution, memory leaks. We've prepared a step‑by‑step guide for setting up a production cluster on 3 nodes — from role configuration to ILM. Our experience shows that a properly configured cluster pays for itself after just a month under load.

Why a Three‑Node Cluster?

A single node is only suitable for development. Production requires a cluster for fault tolerance, horizontal scaling, and load isolation. A 3‑node cluster is 10 times more reliable than a single instance: it survives the loss of one node without data loss and handles parallel queries efficiently. For most web applications, this is the optimal balance of cost and performance.

Node Roles

In Elasticsearch 8.x each node can perform multiple roles. For small clusters (3–5 nodes) all nodes typically combine all roles. For clusters of 10+ nodes, separation is mandatory.

Master‑eligible node – participates in master election, manages cluster state. Minimum 3 master‑eligible nodes for quorum.

Data node – stores shards, executes searches. Most resource‑intensive: needs fast disks (NVMe) and plenty of RAM for JVM heap and OS file cache.

Coordinating node (client node) – receives client requests, distributes to data nodes, collects results. Relieves data nodes during the scatter‑gather phase.

Ingest node – processes documents through ingest pipelines (parsing, enrichment, transformation). Role configuration in elasticsearch.yml:

# Master-only node
node.roles: [ master ]
# Data node
node.roles: [ data, data_content, data_hot, data_warm, data_cold ]
# Coordinating only node
node.roles: []

Minimum Production Configuration: 3 Nodes

All three nodes are master‑eligible + data. This provides quorum (2 out of 3) and data storage. elasticsearch.yml for node 1:

cluster.name: myapp-prod
node.name: es-node-01
node.roles: [master, data, ingest]
network.host: 0.0.0.0
http.port: 9200
transport.port: 9300
discovery.seed_hosts:
  - es-node-01:9300
  - es-node-02:9300
  - es-node-03:9300
cluster.initial_master_nodes:
  - es-node-01
  - es-node-02
  - es-node-03
path.data: /var/lib/elasticsearch
path.logs: /var/log/elasticsearch
xpack.security.enabled: true
xpack.security.transport.ssl.enabled: true
xpack.security.transport.ssl.keystore.path: elastic-certificates.p12
xpack.security.transport.ssl.truststore.path: elastic-certificates.p12

On nodes 2 and 3 only node.name changes.

cluster.initial_master_nodes is used only on the first cluster boot. After the cluster forms, comment out this line — otherwise a restart might cause split‑brain.

JVM and Memory

Elasticsearch defaults to 1 GB heap — critically low for production. Rule: heap = half of available RAM, but not more than 31 GB (above 32 GB JVM loses compressed oops).

In /etc/elasticsearch/jvm.options.d/heap.options:

-Xms16g -Xmx16g 

The remaining memory goes to OS file cache — Elasticsearch uses mmap heavily for reading Lucene segments. On a 64 GB RAM server: 31 GB heap + 30+ GB OS cache is optimal.

System tuning:

# /etc/sysctl.conf
vm.max_map_count=262144
vm.swappiness=1

# /etc/security/limits.conf
elasticsearch soft memlock unlimited
elasticsearch hard memlock unlimited
elasticsearch soft nofile 65536
elasticsearch hard nofile 65536

TLS Certificate Generation and Security Setup

# Generate CA and cluster certificates
/usr/share/elasticsearch/bin/elasticsearch-certutil ca --out /etc/elasticsearch/elastic-ca.p12
/usr/share/elasticsearch/bin/elasticsearch-certutil cert \
  --ca /etc/elasticsearch/elastic-ca.p12 \
  --out /etc/elasticsearch/elastic-certificates.p12

# Set elastic user password
/usr/share/elasticsearch/bin/elasticsearch-setup-passwords auto

HTTP TLS (for client connections) uses a separate certificate:

xpack.security.http.ssl.enabled: true xpack.security.http.ssl.keystore.path: http.p12 

How to Configure ILM to Save Resources?

For logs and temporary data, an ILM policy is mandatory. Without it, indices grow forever and fill the disk. Example policy with four phases:

PUT _ilm/policy/logs-policy
{
  "policy": {
    "phases": {
      "hot": {
        "min_age": "0ms",
        "actions": {
          "rollover": {
            "max_age": "7d",
            "max_size": "50gb"
          },
          "set_priority": {
            "priority": 100
          }
        }
      },
      "warm": {
        "min_age": "7d",
        "actions": {
          "shrink": {
            "number_of_shards": 1
          },
          "forcemerge": {
            "max_num_segments": 1
          },
          "set_priority": {
            "priority": 50
          }
        }
      },
      "cold": {
        "min_age": "30d",
        "actions": {
          "freeze": {},
          "set_priority": {
            "priority": 0
          }
        }
      },
      "delete": {
        "min_age": "90d",
        "actions": {
          "delete": {}
        }
      }
    }
  }
}

Step‑by‑Step Cluster Setup

1. Analysis: determine load, data volume, required node roles.
2. Design: choose number of nodes, disk size, JVM settings.
3. Installation: deploy nodes on servers or containers, configure network.
4. Security configuration: generate TLS, set passwords.
5. ILM and templates: create index lifecycle policies.
6. Testing: verify cluster health, shard distribution, fault tolerance.
7. Deployment: connect the application, enable monitoring via Kibana.

What’s Included in Turnkey Setup

• Cluster deployment on bare metal or cloud
• TLS and basic security configuration
• ILM policy and index template setup
• Integration with Kibana for monitoring
• Operations documentation
• Team training on basic administration
• 2 weeks of post‑launch support

Checking Cluster Health

# Cluster status (green/yellow/red)
curl -u elastic:changeme http://localhost:9200/_cluster/health?pretty

# Node list
curl -u elastic:changeme http://localhost:9200/_cat/nodes?v

# Unassigned shards and reasons
curl -u elastic:changeme "http://localhost:9200/_cluster/allocation/explain?pretty"

Status yellow means all primary shards are assigned but some replicas are not. On a 1‑node cluster this is normal. On a 3‑node cluster yellow indicates a problem.

Connecting from an Application

use Elastic\Elasticsearch\ClientBuilder;

$client = ClientBuilder::create()
    ->setHosts(['https://es-node-01:9200', 'https://es-node-02:9200', 'https://es-node-03:9200'])
    ->setBasicAuthentication('elastic', 'changeme')
    ->setCABundle('/path/to/ca.crt')
    ->build();

from elasticsearch import Elasticsearch
es = Elasticsearch(
    ['https://es-node-01:9200', 'https://es-node-02:9200'],
    basic_auth=('elastic', 'changeme'),
    ca_certs='/path/to/ca.crt',
    retry_on_timeout=True,
    max_retries=3,
)