Azure Cosmos DB: where default configuration cannot handle the load
Azure Cosmos DB forgives a lot in a single-developer environment and almost immediately starts getting in the way in production.
The default configuration assumes that you will choose the partition key deliberately and allocate capacity for the real request profile — if you do not, the first thing that appears is a hot partition: one logical group takes a disproportionately large number of operations, and requests to it begin returning failures due to insufficient throughput, even though the overall reserve seems to be there.
Most often, the partition key mistake originates from the data model, not from the configuration.
If the key is a low-cardinality field or a field used for filtering in reports, the query becomes cross-partition: it queries all partitions, consumes request units in batches, and its time grows along with the collection size.
The symptoms are recognizable — unstable request latency at peaks, a noticeable spread between fast and slow calls, a "sawtooth" in capacity consumption that forces you to keep a reserve above the calculated one.
A separate story is manual edits in the console: indexes, consistency levels, time-to-live policies, and scaling profiles set "quickly" during an incident.
It works, but the environment stops being reproducible: test and prod diverge, deploying a new copy does not produce the expected behavior, and the next edit is again done by hand.
At the same time, capacity limits are not cured by raising the ceiling: without normalizing the partition key and indexes, what grows is not throughput but the per-operation cost.
Next, we break down how this is resolved — from choosing the key and capacity mode to describing the configuration as code, so that environments are assembled identically.
What a well-tuned storage configuration provides
A well-tuned Azure Cosmos DB configuration is not a one-time launch but a set of parameters, each visible in metrics: partition key, indexing policy, consistency level, throughput reserve.
Their combination determines how many operations pass without retries and how quickly a response is returned.
Reducing the failure rate during peak bursts
A peak burst breaks not the storage but the load distribution: one hot partition key collects the majority of writes and begins returning failures due to exceeding throughput.
We align the key with real access patterns, add reserve and autoscaling for the daily profile — the failure rate at peaks drops to tenths of a percent, and a retry avalanche does not start.
Predictable response for reads and writes
Read latency is created not by the network but by scanning: a query without a partition key sifts through extra documents, and response time grows with data volume. Point reads by key and covering indexes for specific filters make the response steady.
Indexing and capacity consumption are directly related: a policy for the needed paths instead of "index everything" reduces write cost, and the freed resource goes to useful workload.
What we record in the final report:
- failure rate at peaks and during off-peak hours;
- read and write response time for key query patterns;
- share of operations completed without retries;
- spread of response times between working hours and peak.
Consistency levels and capacity models
The consistency level and capacity model in Azure Cosmos DB are chosen together with the partition key: sharding depends on the key, and sharding determines how autoscaling behaves at peaks. Changing these decisions after launch takes more time than discussing them at the start.
Consistency: what we lose and what we gain
The stricter the consistency level, the higher the read latency and replication load, but the application does not see stale data. The level is selected per operation: calculations need strict reads, while feeds and counters live on session or eventual consistency.
| Option | When it fits | Limitations |
|---|---|---|
| Strong | Calculations where divergence is unacceptable | Highest read latency |
| Bounded staleness | Read lag within a specified window is acceptable | The window is controlled by the application |
| Session | Reading your own writes | Order between sessions is not guaranteed |
| Provisioned capacity | Steady predictable flow | Reserve is not released on its own, manual scaling |
| Elastic capacity | Bursty load and peaks | Scaling responds with a delay |
Provisioned capacity vs elastic
Provisioned capacity maintains a constant reserve and suits services with steady traffic; elastic adjusts itself, but its reaction to a peak is delayed.
We calculate the load profile from real requests and choose the partition key so that sharding distributes data evenly and no hot partitions appear.
How we carry out work in Azure: from audit to handover
At the start, we take inventory of containers, current partition keys, and the load profile: how data grows, where queries read multiple partitions, and which consumption peaks fall during reporting hours.
In parallel, we record requirements — acceptable downtime window, recovery depth, and who owns the infrastructure after handover.
What we clarify at the start
We need the same inputs as your team: a map of queries and their frequencies, data volume and growth, consistency constraints, the usual deployment method, and the access policy for secrets. Without this, the partition scheme is designed at random.
- Partition scheme audit. We analyze partition keys, indexes, and the query profile: where reads go across all partitions, which ones hit the ceiling at peak. The output is a map of bottlenecks and a verdict on the layout.
- Partitioning and consistency level design. We select a key for real queries, check partition growth, and choose the consistency level — it affects latency and the allowable divergence between replicas.
- Infrastructure as code. We describe containers, throughput, indexes, and access roles declaratively: the configuration lives in a repository, passes review, and is deployed identically across multiple environments.
- Cosmos DB configuration via Terraform and zero-downtime migration. We bring up the new layout alongside the production one, switch traffic in stages, and transfer data with lag monitoring. The downtime window is zero, and rollback to the previous scheme remains ready.
- Load profile validation. We run peak and sustained scenarios, observe behavior as partitions grow, and set up alerts and scaling rules. We document the backup policy in writing and verify recovery.
- Handover and team training. We hand over the repository, environment schema, access policies, and change process, and conduct a walkthrough with engineers — operations continue without us.
What remains with you after implementation: configurations and policies
We do not leave the infrastructure as a "black box": along with the working cluster, you receive artifacts by which it can be developed and operated by your own efforts.
The description is stored in your repository under version control — changes go through review, not manual edits in the console.
Composition of the documentation handed over
- Configuration as code. Modules for describing Cosmos DB databases, accounts, consistency level, and backup policies, with a separate set of variables for test and production environments.
- Partitioning and index schemas. A partition key map with rationale for the choice, per-container indexing policies, and throughput calculation for the load profile.
- Access policies and roles. A "who can do what" matrix, binding of permissions to service accounts and managed identities, and the principle of least privilege.
- Schemas and diagrams. Deployment topology, cross-region replication, list of failure points, and traffic switchover procedure.
- Backup policy. Snapshot frequency, retention periods, and a step-by-step recovery scenario with data integrity verification.
- Incident runbook. Typical failures — throughput exhaustion, partial request failures, growing latency: what to look at and which parameter to change.
- Operations policies. Change release procedure, maintenance windows, request cost control, and rules for rolling out new containers.
Each document is tied to the actual state of the environment and updated along with it: a policy detached from the code quickly diverges from reality.
Case study: rebuilding the partition scheme for peak load
A multi-tenant service: orders, warehouse, reports. Containers in Azure Cosmos DB grew with the number of clients, and during seasonal peaks requests began to fail. We analyzed not the load but the partition scheme: it had once been designed for steady flow, not for peak.
Before
The partition key was the client identifier. For large tenants, tens of millions of documents accumulated in one partition, while for others partitions sat idle — some physical partitions operated at their limit.
A bulk export for a large client pushed consumption to the ceiling, and the database responded with failures for exceeding the limit: at peak, the share of unsuccessful requests reached several percent.
A separate problem was that all fields in a row were indexed, and every write pulled an index rebuild.
After
We switched to a composite key: tenant plus time window — a month for operational documents, a year for archive. Clients whose window remained hot were split with a synthetic suffix of eight segments from the identifier hash.
Reports were moved to a separate container with its own key, indexing was narrowed to fields of real filters, and time-to-live was enabled for archival history.
The share of failures due to limits at peak dropped from several percent to tenths of a percent, response time for heavy queries decreased severalfold, and the load spread between hot and cold partitions came within one order of magnitude instead of two. There was three times more data at the same throughput.
A class of failures disappeared where one tenant consumed a partition and degraded response for everyone. The scheme transfers to any multi-tenant model: the partition key is chosen by the access pattern, not by the entity structure.
What to do if the database is already deployed and the load is growing?
Rebuilding the partition scheme on live data is possible — but not "in place." The partition key determines the physical placement of documents, so it cannot be changed in an existing container: a new container is brought up and data is transferred into it while the already deployed Cosmos DB database continues serving traffic. This happens without stopping the service if the application can work with two keys simultaneously.
The procedure is: capture the load profile and data distribution by the current key, choose a new one by cardinality and uniformity, and enable dual writes to the target container.
Then a background transfer with checksum verification by logical partitions, gradual switching of readers, and only after reconciliation — decommissioning the old circuit.
Application backward compatibility is built in advance: the access layer reads by both keys and writes to the new one, so rollback takes minutes, not hours.
Switching between capacity models does not go as smoothly — changing the mode affects the account level, so a short maintenance window is planned with warm-up and verification of key scenarios immediately afterward.
What rebuilding the partition scheme provides: cross-partition queries stop consuming shared throughput, the most loaded logical partitions no longer hit the ceiling, and data growth is distributed evenly again.
We start with measurements, not with switching — then the decision relies on the load profile, not on guesses.
Contested points: consistency, indexes, access
Almost every Cosmos DB decision is a trade-off between latency, availability, and request unit consumption. The same doubts come up in pre-launch reviews — below are short answers to those that most often slow down approval.
How strict should consistency be?
Strong gives linear operations but ties reads to the write region: latency grows and availability drops if a region is lost. Session level covers most scenarios — the client reads its own writes. We keep it as the default and enable strong selectively.
Is indexing all fields normal?
No: every extra path adds request unit consumption for writes and bloats storage. We exclude fields without filters and sorts, keep composite indexes for specific query patterns, and check the policy against the query log before going live.
Can we avoid using the master key in the application?
Yes, through managed identities and roles — built-in or custom, scoped to the account, database, or collection. The key remains in the secret store only for migrations and is rotated on a schedule.
What does backup provide and how do we restore?
Periodic copies with a retention window or continuous mode. Restore is performed not in place but into a new account: we configure the connection string and access again, so we rehearse the switchover in advance.
Who consumes the request processing budget?
Most often it is not the source but the "hot" sections of partitions: when the partition key is skewed, almost the entire flow goes to one part. We look at metrics by sections, profile heavy queries, and redistribute the key or move reads to a separate model.
Let's discuss the task and come back with a work plan
So that the work estimate relies on facts rather than assumptions, we will collect an input data set. Below is what to send before the call; with limited access to production, some items are replaced with anonymized exports.
- Data schema: containers, partition keys, current indexes and TTL, consistency model, collection volumes and growth rate.
- Load profile: requests per second and read/write ratio, document size, peak windows, client geography, and regional distribution.
- Current configurations: throughput mode, consistency levels, indexing policies, backup settings, infrastructure descriptions and state if resources are already managed by Terraform.
- Access for audit: read access to the subscription, metrics, diagnostic logs, and slow query logs; anonymized exports are sufficient.
- Requirements and constraints: target recovery time and acceptable data loss point, regulatory norms, maintenance windows, change approval procedure.
- Acceptance criteria: 95th percentile latency, share of requests without throughput throttling, predictability of storage growth.
On the call, we will review this data and come back with a work estimate, a phased implementation plan with rollback points, a documentation handover procedure, and post-launch support options.