Apache Cassandra & ScyllaDB Administration: Audit & Support

Cassandra and ScyllaDB are databases for tasks where PostgreSQL and MySQL give up: millions of writes per second, petabytes of data, zero tolerance for downtime. But the price for this linear scalability is an order of magnitude higher operational complexity. We often encounter situations where a poorly chosen partition key or a stray ALLOW FILTERING in production degrades an entire cluster. Over years of work we've developed a methodology to prevent such scenarios. More about Apache Cassandra can be read in the official documentation.

Key Differences Between ScyllaDB and Cassandra

ScyllaDB is written in C++ and implements a shard-per-core architecture: each CPU core processes its own dataset with no shared state between threads. This gives 10x lower p99 latency under the same load compared to JVM-based Cassandra. In practice: what required 6 Cassandra nodes runs on 2 ScyllaDB nodes with better performance.

Cassandra is a mature product with a rich ecosystem, better compatibility with enterprise tools, and a more conservative release nature. ScyllaDB is compatible with CQL and Cassandra drivers but has its own specifics (e.g., different implementation of lightweight transactions). Our experience shows that the choice depends on latency requirements and node density.

Cluster Topology and Configuration

Minimum production cluster: 3 nodes with RF=3. This means each partition is replicated to all three nodes, and losing one node does not lose data. We recommend instances with dedicated CPUs and SSDs for the commitlog.

cassandra.yaml for a production node:

cluster_name: 'prod-cluster'
num_tokens: 256
seed_provider:
  - class_name: org.apache.cassandra.locator.SimpleSeedProvider
    parameters:
      - seeds: "10.0.1.10,10.0.1.11"
listen_address: 10.0.1.10
rpc_address: 0.0.0.0
broadcast_rpc_address: 10.0.1.10
endpoint_snitch: GossipingPropertyFileSnitch
auto_bootstrap: true
# Compaction
compaction_throughput_mb_per_sec: 64
# Memtable
memtable_allocation_type: offheap_objects
memtable_heap_space_in_mb: 2048
# Commitlog
commitlog_sync: periodic
commitlog_sync_period_in_ms: 10000
commitlog_segment_size_in_mb: 64
concurrent_reads: 32
concurrent_writes: 32
concurrent_counter_writes: 16

GossipingPropertyFileSnitch is used in production — it reads cassandra-rackdc.properties to determine the datacenter and rack:

dc=DC1 rack=RAC1 

Why Choosing the Right Partition Key Matters

Partition key determines on which node the data is stored. If the key is poorly chosen, a hot partition occurs — one node gets 80% of the load while others idle. Cassandra is not relational: no JOINs, no secondary indexes in the traditional sense. Schema is designed for queries, not normalization.

Example of a proper table for storing events sorted by time:

CREATE TABLE events_by_user (
    user_id UUID,
    event_time TIMEUUID,
    event_type TEXT,
    payload TEXT,
    PRIMARY KEY (user_id, event_time)
) WITH CLUSTERING ORDER BY (event_time DESC)
    AND compaction = {
        'class': 'TimeWindowCompactionStrategy',
        'compaction_window_size': '1',
        'compaction_window_unit': 'DAYS'
    };

user_id is the partition key; data of one user is stored together. event_time is the clustering key, determining order within a partition. Query WHERE user_id = ? AND event_time > ? executes efficiently without ALLOW FILTERING.

Configuring Compaction for High Load

Compactions repack SSTables and remove deleted records. A typical mistake is choosing the wrong strategy. For time-series data use TimeWindowCompactionStrategy (TWCS): it merges SSTables only within a window, reducing load. For random access — SizeTieredCompactionStrategy (STCS), but watch the min_threshold (default 4) — too frequent compacting kills IO. For ScyllaDB additionally configure scylla.yaml:

compaction_static_shares: 0.5 table_compaction_dynamic_shares: true 

The compaction_throughput_mb_per_sec parameter in cassandra.yaml limits compaction throughput. We recommend setting 64-128 MB/s to avoid impacting read/write latency.

Cluster Maintenance: nodetool and Monitoring

nodetool repair is a critical operation often ignored. Without regular repair, replicas diverge. Run every gc_grace_seconds (default 10 days). Use incremental repair to reduce load:

# Incremental repair — less load on cluster
nodetool repair -inc -pr keyspace_name table_name

# Monitor node status
nodetool status
nodetool tpstats  # thread pool statistics
nodetool compactionstats  # current compaction tasks

ScyllaDB adds scyllatop — interactive top for shard-level monitoring. For Prometheus: cassandra-exporter or built-in JMX exporter for Cassandra, scylla-monitoring stack for ScyllaDB (includes ready-made Grafana dashboards). We set up alerts on p99 latency, compaction backlog, and repair status.

ScyllaDB Specifics

In ScyllaDB there is no separate nodetool repair — instead scylladb-manager (a separate service) manages repair tasks in the background, distributing load over time. This is much more convenient. scylla.yaml has additional parameters:

---
# Developer mode (disable in production!)
developer_mode: false
# SMP — number of threads (equal to core count)
smp: 8
# Overprovisioning (for VMs)
overprovisioned: false
---

What Our Administration Service Includes

• Audit of existing cluster: analyze nodetool status, data distribution (nodetool tablestats), top loaded tables, identify ALLOW FILTERING in logs, compaction state.
• Deploy new cluster: Ansible role for installation, config tuning, seed node initialization, create keyspaces with optimal RF, load test via cassandra-stress.
• Deliver results: complete runbook with instructions for adding/replacing nodes, repair schedule, Prometheus alert configuration, team training.
• Guarantee: we support the cluster for 30 days after handover, fixing any issues. Our experience: 10+ years in production, 50+ successful projects.

Cost is calculated individually after an audit of load and data volume requirements. For a consultation or to order an audit, contact us — we will select the optimal plan.

Want to secure your cluster? Get an engineer consultation — we'll audit and provide improvement recommendations. Order your cluster audit today — contact us to schedule.