Apache Cassandra Setup for High-Load Web Applications

Setting up Apache Cassandra for a web application involves more than just installing a package. Without proper schema design and cluster configuration, you will face slow queries, hot partitions, and unstable operation under load. Over 5 years, we've configured Cassandra for 30+ projects: event streams, metrics, logs. The most common mistake is an incorrect primary key. Ignoring compaction and insufficient memory for memtables leads to 2–3x performance degradation. Proper Cassandra setup from the start saves weeks of rework. Here are our proven techniques.

Where Cassandra is indispensable

Time series with millions of events per second, activity feeds, logging systems, IoT telemetry — scenarios requiring fast, high-volume writes. Netflix, Discord, Apple use Cassandra precisely for this. Discord stores trillions of messages. For OLTP with complex transactions, it is not suitable. We use Cassandra for storing metrics and logs: it provides linear write scalability and fault tolerance without a single point of failure.

How to install and configure Cassandra 4.1

Installation of Cassandra 4.1:

echo "deb https://debian.cassandra.apache.org 41x main" > /etc/apt/sources.list.d/cassandra.sources.list
curl https://downloads.apache.org/cassandra/KEYS | apt-key add -
apt update && apt install -y cassandra

After installation, edit cassandra.yaml:

---
cluster_name: 'MyAppCluster' # Network
listen_address: 10.0.0.1
rpc_address: 10.0.0.1
seeds: "10.0.0.1,10.0.0.2,10.0.0.3"
# Directories
data_file_directories:
  - /var/lib/cassandra/data
commitlog_directory: /var/lib/cassandra/commitlog # separate disk for speed
hints_directory: /var/lib/cassandra/hints
saved_caches_directory: /var/lib/cassandra/saved_caches
# Performance
concurrent_reads: 32
concurrent_writes: 32
concurrent_counter_writes: 16
memtable_heap_space: 2048
compaction_throughput_mb_per_sec: 64
# Replication and consistency
endpoint_snitch: GossipingPropertyFileSnitch
# JVM (in jvm11-server.options)
num_tokens: 256
---

Additionally, configure JVM to avoid long GC pauses:

# /etc/cassandra/jvm11-server.options
-Xms8G
-Xmx8G
-XX:+UseG1GC
-XX:G1RSetUpdatingPauseTimePercent=5
-XX:MaxGCPauseMillis=300
-XX:InitiatingHeapOccupancyPercent=70

Comparison of compaction strategies

Choosing compaction directly affects read and write performance. For example, TimeWindowCompactionStrategy reduces the number of SSTables by 2–3x compared to SizeTieredCompactionStrategy when working with time series. We recommend TimeWindowCompactionStrategy for data with TTL, and LeveledCompactionStrategy for scenarios with intense updates.

Which compaction strategy to choose?

Base your choice on the data characteristics. If data has TTL and is written at a constant rate — TimeWindowCompactionStrategy. For frequent updates and deletes — LeveledCompactionStrategy. SizeTieredCompactionStrategy is a safe choice if unsure.

Why proper partition key selection matters?

The partition key determines data distribution across nodes. If chosen incorrectly, some nodes will be overloaded while others sit idle. For example, for the user_events table we use user_id — this guarantees even distribution. For time series (table metrics), a composite key (service, bucket, metric_name) groups metrics by service and time window. Never use columns with few unique values (boolean or enum) — this leads to hot partitions.

Data schema — Query-driven design

-- Keyspace with replication
CREATE KEYSPACE myapp
WITH replication = { 'class': 'NetworkTopologyStrategy', 'dc1': 3 }
AND durable_writes = true;

USE myapp;

-- User event feed
CREATE TABLE user_events (
    user_id uuid,
    occurred_at timestamp,
    event_id uuid,
    event_type text,
    payload text,
    PRIMARY KEY ((user_id), occurred_at, event_id)
) WITH CLUSTERING ORDER BY (occurred_at DESC)
AND compaction = {'class': 'TimeWindowCompactionStrategy', 'compaction_window_unit': 'DAYS', 'compaction_window_size': 7}
AND default_time_to_live = 7776000;

-- User statistics
CREATE TABLE user_stats (
    user_id uuid PRIMARY KEY,
    total_orders counter,
    total_spent counter,
    last_active timestamp
);

-- Time series metrics
CREATE TABLE metrics (
    service text,
    bucket timestamp,
    metric_name text,
    ts timestamp,
    value double,
    PRIMARY KEY ((service, bucket, metric_name), ts)
) WITH CLUSTERING ORDER BY (ts DESC)
AND compaction = {'class': 'TimeWindowCompactionStrategy', 'compaction_window_unit': 'HOURS', 'compaction_window_size': 1};

Integration with Node.js backend

import cassandra from 'cassandra-driver'

const client = new cassandra.Client({
  contactPoints: ['10.0.0.1', '10.0.0.2', '10.0.0.3'],
  localDataCenter: 'dc1',
  keyspace: 'myapp',
  credentials: {
    username: 'cassandra',
    password: process.env.CASSANDRA_PASSWORD!
  },
  pooling: {
    coreConnectionsPerHost: {
      [cassandra.types.distance.local]: 3,
      [cassandra.types.distance.remote]: 1
    }
  },
  socketOptions: {
    readTimeout: 12000
  }
})

await client.connect()

const insertEvent = await client.prepare(`
  INSERT INTO user_events (user_id, occurred_at, event_id, event_type, payload)
  VALUES (?, ?, ?, ?, ?)
`)

const selectEvents = await client.prepare(`
  SELECT * FROM user_events
  WHERE user_id = ? AND occurred_at >= ? AND occurred_at <= ?
  ORDER BY occurred_at DESC
  LIMIT ?
`)

async function writeEvents(events: UserEvent[]) {
  const batch = events.map(e => ({
    query: insertEvent,
    params: [
      cassandra.types.Uuid.fromString(e.userId),
      new Date(e.occurredAt),
      cassandra.types.TimeUuid.now(),
      e.eventType,
      JSON.stringify(e.payload)
    ]
  }))
  await client.batch(batch, { prepare: true, logged: false })
}

async function* fetchEvents(userId: string, from: Date, to: Date) {
  const options = { prepare: true, fetchSize: 1000 }
  let pageState: Buffer | undefined
  do {
    const result = await client.execute(
      selectEvents,
      [cassandra.types.Uuid.fromString(userId), from, to, 1000],
      { ...options, pageState }
    )
    yield result.rows
    pageState = result.pageState as Buffer | undefined
  } while (pageState)
}

Consistency levels

const { types: { consistencies } } = cassandra
await client.execute(insertEvent, params, { consistency: consistencies.localQuorum })
await client.execute(selectEvents, params, { consistency: consistencies.one })
await client.execute(criticalQuery, params, { consistency: consistencies.quorum })

Monitoring and diagnostics

nodetool status nodetool tpstats nodetool cfstats myapp.user_events nodetool compactionstats nodetool cleanup myapp 

Enable slow query logging in cassandra.yaml: slow_query_log_timeout_in_ms: 500.

Typical mistakes and their solutions

• Hot partitions due to wrong key: use composite keys with high cardinality. Avoid columns with few values.
• High memory consumption: reduce memtable_heap_space or switch from G1GC to ParallelGC.
• Slow writes: check the disk for commitlog — use a separate SSD. Increase concurrent_writes.

How long does a full setup take?

A typical project with three nodes and integration takes 2–3 weeks. It includes requirements audit, schema design, cluster deployment, configuration tuning, and backend adapter writing. Complex clusters with multiple data centers may take up to 5 weeks. Get an accurate estimate for your project — contact us.

Our cases and guarantees

We have completed over 30 projects with Cassandra, including metric systems for an advertising platform (100 million events per day) and activity feeds for a SaaS service. In every project, we guarantee stable cluster operation for one month after delivery. Our engineers have 5+ years of experience with Cassandra and related technologies. Contact us for a consultation — we will discuss your architecture.

What is included in turnkey setup

• Audit of current architecture and load requirements
• Data schema design (query-driven design)
• Cluster deployment (bare-metal / cloud / Docker)
• Optimization of cassandra.yaml, JVM, and network settings
• Backend integration (Node.js, Python, Go)
• Schema documentation and operation instructions
• Team training and operational recommendations
• Guaranteed stable cluster operation for one month after delivery

Order Cassandra tuning for your project — get a consultation from an engineer with 5 years of experience.