MongoDB WiredTiger Configuration: Cache, Compression, and Monitoring

Configuring MongoDB WiredTiger when the database hits memory and disk limits

When the database hits memory and disk limits, degradation begins unnoticed: the WiredTiger cache stops accommodating the working set, and an ever larger share of reads bypasses it — first to fast storage, then deeper.

While there is headroom, metrics stay flat, but once the active data set exceeds the available cache, eviction of data to disk can no longer keep up with the stream of changes.

The database does not go down — it slows down, and this is the most unpleasant class of failure: formally everything is alive, in practice the service cannot sustain the load.

The first thing to show is growth in database response latency: requests queue up on modified pages, which background threads cannot flush fast enough.

Latency grows nonlinearly — with a slight excess of the working set over the cache, response degrades by multiples, not by percentages.

Then feedback kicks in: the application accumulates connections and re-reads the same data more aggressively, increasing pressure on the cache further.

Next, replicas diverge. Secondary nodes replicate the primary's operation stream, and as soon as the primary can no longer keep up with the load, replica lag grows visibly.

Reads from replicas begin serving stale data, and reporting and background jobs work from an incomplete picture. In essence, one node that has hit memory limits drags the entire cluster with it.

Simply "adding memory" does not remove the problem: without matching cache size to the working set, block compression, eviction policy, and checkpoint frequency, the same limit will be reached again.

That is why configuring MongoDB WiredTiger is work with parameters and monitoring, not a single configuration edit, and below we break down how this is addressed: from diagnostics using the engine's internal counters to choosing a compression scheme for the data profile.

What changes after tuning cache and compression parameters

Tuning parameters is not a one-time configuration edit but testing hypotheses against metrics. We look at three slices: how much of the working set remains in memory, how much space the collection occupies on disk, and how the secondary node behaves under load.

Below are the shifts that show that configuring MongoDB WiredTiger produced results rather than coinciding with a one-off traffic spike.

What we keep on the dashboard

Cache hit ratio, depth of the disk request queue, secondary node lag, compression ratio per collection, and node startup time after restart.

We capture these values before and after changes and store them as graphs: without a baseline, in a month no one will be able to prove that things got better.

Which failures stop occurring

Eviction of pages to disk during peak hours goes away — that is exactly what causes stepwise latency growth that cannot be cured by scaling. Replica lag evens out, so reads from secondary nodes stop serving stale data.

Restart and recovery fit into a predictable window, and collections with constant appends no longer consume disk.

Comparison of approaches: compression, cache, and collection layout

The decision on compression and layout is made from the workload profile, not by default: read share, working set size, data retention period.

WiredTiger allows setting compression separately for the collection and separately for indexes, so the same server can keep hot data with a fast codec and an archive with a dense one.

A mistake in choosing has a bigger impact on read latency than it saves in space: dense compression saves disk but adds CPU work on every read.

Extremes break predictably. If you compress everything as much as possible, the server hits CPU limits, and read latency grows for no reason. If you keep the entire working set in cache, the physical memory limit brings back eviction and misses — just later and less noticeably.

The hybrid option separates collections by purpose: hot ones remain with a fast codec and increased cache, the archive moves to a separate layout with dense compression, and reporting reads the moved data. Settings are reviewed based on metrics rather than set once.

Order of work: from workload profile to handover under control

Work proceeds from measuring the actual load: the same set of WiredTiger parameters produces opposite results on different profiles — in one case a latency gain, in another a throughput drop.

The order below is built so that each change relies on the previous measurement and does not break what has already been achieved.

1. Audit and workload profile. We capture read and write operation statistics, cache hit ratio, slow query duration, peak windows. The output is a load model from which we calculate the rest.
2. Storage schema and indexes. We analyze document structure, index composition and size, fragmentation, placement of data and journal on disks. We remove unnecessary indexes and add missing ones: before tuning parameters, this yields more than the tuning itself.
3. Parameter tuning. We calculate cache size for the working set and available memory, choose the compression algorithm and level for collections and indexes, configure the journal and checkpoint frequency. Each value is justified by measurement, not by a reference guide.
4. Validation under load. We reproduce the profile on a test environment and compare latency, throughput, and memory consumption before and after. We separately observe behavior during cache overflow and at peak — that is where settings show themselves.
5. Observability and thresholds. We expose key metrics, set thresholds and alerts for them so degradation becomes visible before user complaints.
6. Handover of support. We provide configurations, schema description, metric map, and rollback procedure, and go through typical situations with your team.

What remains with you: configurations, schemas, and procedures

We hand over the package as working artifacts, not a folder of files: configurations are stored in version control alongside code, schemas as diagrams, procedures as step-by-step instructions with commands.

Each item has a clear entry point: where to start during an incident and who is responsible on your side.

• Cluster configuration. WiredTiger parameters: cache size for the working set, collection and journal compression, checkpoint intervals; values are parameterized by environment, secrets are separated out.
• Data placement schema. What is sharded and by which keys, what remains on replicas, where the boundary between hot and archival collections is, which indexes must exist.
• Access policies. Roles based on least privilege, separate accounts for applications, replication, and monitoring, rotation procedure.
• Monitoring procedure. Metrics and thresholds, ready-made queries for investigating latency growth, replica lag, and disk usage, escalation procedure.
• Operations documentation. Bringing up a node from scratch, planned switchover, restore from a copy — with time estimates for each step.
• Backup plan. Schedule, retention depth, mandatory restore verification on a test environment.
• Performance baselines. Load profiles and reference values before and after changes — you can see where normal is and where degradation is.

Case: storage growth slowed down, failures under load disappeared

What it was before the changes

A MongoDB cluster served a product catalog: nightly recalculation, a stream of events from applications, and constant reads.

Over six months, data volume grew almost threefold, the storage cache remained at the default value and did not cover the working set — a noticeable share of reads went to disk.

During peak hours this caused failures under load: response time exceeded the target, some operations failed with a timeout error, and the replica lagged behind the master node.

What we did

We recalculated the cache size for the actual working set, enabled block compression for collections, and configured checkpoint intervals so writes occurred in even portions rather than bursts. We moved heavy analytical queries off the primary node.

In parallel, we set up monitoring for four indicators: storage growth, response time, share of disk reads, and number of failures.

Metrics before and after

• storage growth — plus 8% per month at the start, plus 1-2% after tuning;
• share of disk read operations — from ~30% to 3%;
• read response time — reduced by approximately 2.5 times;
• failures under load — isolated cases per day, none over a quarter of observations;
• data file volume — compression produced a threefold reduction, backups are assembled multiple times faster.

We also specifically added an alert in monitoring for working set growth: the next node expansion is planned in advance, not after a failure.

How does WiredTiger differ from other storage engines?

WiredTiger is the default database storage engine in the current branch: it determines how documents are written to disk and read back. There are three differences from engines with collection-level locks: write-ahead log, data snapshots, and in-memory cache. Each defines its own operating mode.

The write-ahead log works like this: a change first goes to a sequential log and only then to data files. Acknowledgments are grouped, so several operations are committed with one disk access.

This provides a clear recovery point after an abnormal shutdown, but the write itself becomes a bottleneck on loaded nodes.

Data snapshots and the concurrency model are the second part. A reader sees a consistent snapshot as of the start of the operation and does not block writers: locks are taken at the document level, not the collection level.

Therefore an analytical query does not slow down writes, and adding cores yields gains — as long as the working set fits in memory.

The in-memory cache is tuned most often. Its size determines whether the working set is served without disk access: with a small cache, disk reads and latency tails grow; with too large a cache, the operating system evicts pages.

Compression also matters: data and indexes are stored compressed, file and I/O volume drops, but CPU load grows.

For operations this means: monitor cache utilization, the queue of journal writes, and replica lag, and limit long transactions so snapshots do not grow.

Technical doubts: memory, compression, indexes, failures

Doubts before changing settings are usually the same: what cache size to choose, what index compression actually provides, how to distinguish cache overflow from ordinary load, and whether it is acceptable to edit parameters while the application is running. Below are short answers based on operational practice and official MongoDB documentation.

Will index compression slow down queries?

Prefix compression reduces the index size on disk and the part kept in memory; decompression falls on the CPU, but it is usually cheaper than reading extra pages. We switch it during a low-load window and monitor response time.

How to distinguish a loaded cache from an overflowing one?

We look at database observability: share of modified pages, eviction wait, cache misses, write latency. If modified pages stay near the upper bound and the application waits on writes, eviction has become the bottleneck. We make decisions based on dynamics over hours, not a single spike.

Can parameters be changed while the application is running?

Some settings apply without restart, while others require a shutdown with a verified sequence of actions. Before changing, we check runtime applicability, record the original values, and prepare a rollback plan — then editing a live database does not turn into downtime.

Let's discuss the task: what we need from you to start

So that the database work estimate is based on facts rather than guesses, a short set of inputs is enough. Storage audit begins with the WiredTiger configuration, a metrics snapshot, and the workload profile — they show where the bottleneck is and what to change first.

• Current parameters: WiredTiger cache size, selected compression mode for collections and indexes, write-ahead log settings.
• Volume ratios: how much space the data set and indexes occupy relative to available memory — this immediately shows whether we are hitting the cache or the disk.
• Metrics snapshot for a representative period: share of page eviction from cache, write queues, read latency, growth in disk accesses.
• Workload profile: peak hours, nightly batch jobs, heavy queries, replication behavior.
• Data schema: which collections grow faster than others, where there are unused or duplicate indexes.
• Topology and constraints: a single node or replica set, whether sharding is present, what maintenance window is acceptable, and what failure requirements exist.
• History of attempts: what has already been changed in settings, what effect it had, and what users complain about.

At the output you receive a work plan with stages, priorities, and a point to return to the original configuration, as well as a database support proposal — regular metric monitoring and parameter adjustment as growth continues. Describe the task and attach what you have from the list: we will return with an estimate and plan.