Why AWS DynamoDB Degrades Under Load
Most often, Amazon DynamoDB degrades not because of data volume but because of traffic shape.
If the partition key is chosen by a "convenient" attribute — status, document type, date without a suffix — the entire flow goes to one hot partition, while the others sit idle.
Then throughput throttling kicks in: some writes are rejected, the client retries them, and the load on that same partition doubles.
The second common source is reading by full table scan instead of point queries by key. Such a request goes through the entire item set, response latency grows linearly with the table, and under peak traffic it competes for the same capacity as production operations.
Externally it looks like "the storage started lagging out of nowhere," even though the schema and data have not changed.
A separate story is indexes that do not exist for real access patterns. Queries go by non-key fields, filtering happens after retrieval, and instead of one access to the needed partition, the system scans dozens.
A secondary index added after the fact requires rebuilding and changes the write profile — it should be designed together with the access schema, not after the first complaints.
Peak spikes finish the picture: write latency grows, operations go into retry, retries pile into an avalanche, and the table stops accepting the load it previously handled.
Let's break down how this is addressed — from partition key choice to capacity mode and hot partition monitoring.
Effect of a Redesigned Data and Access Schema
Changes after schema redesign are visible not in reports but in the behavior of the storage under load. Below is what we measure at acceptance: the same set of scenarios is run before and after, on the same traffic profile.
Below are the effects that AWS DynamoDB delivers specifically through the data schema and access model, not through purchased capacity.
Response Time on Hot Queries
The partition key is chosen so that reads go to one partition instead of a table pass with filtering.
Response time on the "order — payment — status" scenario dropped severalfold, and the gap between a typical and a rare slow response narrowed: heavy tails stopped being the norm, and client-side external timeouts no longer trigger.
Write Failure Class Disappeared
Previously, a seasonal peak led to write failures: operations above the per-partition limit were rejected, and data had to be backfilled manually.
After spreading the load across keys and correctly setting the capacity mode, this class of failures disappeared — peaks pass routinely, without manual catch-up jobs and reconciliation of discrepancies.
Predictable Load and Throughput Headroom
Capacity is configured for target utilization with automatic scaling based on actual usage and a buffer for spikes.
The consumption graph stopped being sawtooth, throughput headroom is visible in advance, and adding load no longer requires manual intervention at the moment of peak — only a planned review of target values.
Data Volume Growth Without Storage Redesign
Data is laid out by composite keys, and old records expire according to the retention policy. Tables grow linearly, so data volume growth does not require schema migration, service downtime, or a separate migration project — a new season simply adds records.
AWS DynamoDB Operating Modes and When They Fit
The load profile decides more than the table schema: the same data set behaves differently on a steady stream and on sharp peaks.
AWS DynamoDB separates this into independent settings — capacity mode, secondary indexes, cross-region replication, caching layer, change stream — and their combination determines where read failures and extra disk accesses will appear.
Comparison of Options
| Option | When It Fits | Limitations |
|---|---|---|
| On-demand capacity | Unpredictable traffic, product launch | More expensive under steady high load, no manual headroom |
| Provisioned capacity with autoscaling | Predictable schedule, stable stream | Requires thresholds and monitoring; throttling is possible on a sharp spike |
| Secondary index (local or global) | Different access key, queries by attributes | Its own capacity, limited number of indexes and key size |
| Global tables | Reads close to the user, protection from region failure | Replication latency, conflicts when writing to both regions |
| Change stream | Denormalization, export, reacting to edits | Limited retention window, ordering only within a key |
| Caching layer | Hot repeated reads | Invalidation, risk of stale data, extra node |
In practice, layers are combined: provisioned capacity for a steady stream, a global index for queries by attributes, a change stream for synchronization, cache for the hottest reads.
Before choosing, we capture the profile: read-to-write ratio, distribution across keys, share of queries not by the primary key.
Work Order: Data Schema and Access Key Audit
An audit before changes is cheaper than rebuilding tables after launch: in DynamoDB the schema is embedded in the keys, and on live data it can only be changed by migration.
Therefore, we first analyze how the application actually accesses the storage, and only then touch the settings.
- Capture the query profile — using logs and code, we look at which accesses occur, at what frequency, what goes into the query condition, and what is returned in full.
- Analyze the current schema: check whether the partition key distributes load evenly and whether the sort key returns the needed ranges without a full scan.
- Design indexes for specific query patterns: local — within one partition, global — for the rest, making sure each index does not make writes more expensive.
- Configure permissions: a separate role for each service, only the necessary actions and only the needed tables and indexes, without shared access keys in configuration.
- Run a load test: execute real patterns at the target volume, review latencies and error rate at peak, and behavior when provisioned capacity is exhausted.
- Document the procedure and hand it over to the team: table and index schemas, role descriptions, naming rules, monitoring points, and the schema change procedure.
After that, the team adds new query patterns based on ready keys and indexes, instead of rebuilding the table structure.
Table Schemas, Access Policies, and Procedures
After launching storage on DynamoDB, the customer's team is left not with a set of screenshots but with a set of artifacts: using them, a new engineer can bring up tables in another environment and understand why the access model is designed the way it is.
Documents live in the repository alongside code and change together with the infrastructure, not as a separate file on disk.
- Table and index descriptions — partition and sort key, composition of secondary indexes, and rationale for the choice: new queries do not lead to a full database scan.
- Roles and access policies — separate roles for services, engineers, and build pipelines, least-privilege permissions, separation of reads and writes between environments.
- Encryption settings — what is encrypted at rest and in transit, where keys are stored, who is responsible for their rotation.
- Replication scheme — which regions receive a copy of the data, what the propagation latency is, and by what rule traffic switches during a failure.
- Backup and recovery procedure — snapshot schedule, retention depth, regular recovery testing on a test environment with time measurement.
- Incident handling procedure — monitored latency and failure metrics, alert thresholds, responsible parties, and escalation procedure.
- Schema change log — how attributes and indexes are added without stopping writes, and who approves the change.
Case Study: Peak Load Without Write Failures
Before: Partition Skew and Failures at Peak
Write load came in waves: nightly batch jobs and daytime order spikes hit the same range of keys. Some partitions of the AWS DynamoDB table constantly hit the throughput ceiling while neighboring ones sat idle.
At peak, this resulted in write failures and retries, and tail write response time reached 900 ms — the frontend showed checkout errors, and orders had to be completed manually.
After: Smoothed Load and 3x Volume Growth
First, the data model was reworked: the hot key was split into a composite one with a "salt" suffix so that writes spread across partitions, and heavy analytics queries were moved to sparse indexes.
Then the table was switched to automatic scaling based on actual consumption and the change stream was enabled — background reads stopped competing with application queries for the same partitions.
Separately, we configured collection of consumption metrics per partition and an alert for growth in retry count, so skew would be visible before peak, not after.
Tail write response time dropped from 900 to 140 ms, the share of write failures at peak — from 3.2% to zero, and the spread in consumption between partitions went from tenfold to twofold.
Data volume grew 3x over the same period: without changing the platform, without manual partitioning, and without rewriting the application.
The failure class "hit the limit on a hot key" is closed at the data schema level, so it returns only when the load profile changes — then we repeat the key distribution check.
What to Do If the Table Is Already Deployed and Has Data
In DynamoDB, the partition key and sort key are fixed when the table is created, and they cannot be changed in place — schema migration always means moving data to a new table.
This does not block the service: while the copy is being filled, the application continues to work with the old table, and the new one exists in parallel.
Zero-downtime migration is built on two streams at once. The new table is created alongside, and the application gets dual writes: every change goes to both the old and new table, so divergence does not accumulate.
Historical data is transferred in batches with cursor continuation — the process is idempotent, it can be interrupted midway and restarted without duplicates, which is important at large volumes.
Backward compatibility rests on the access layer, not on the write code. Reads first go to the new table, and if the record is absent — to the old one, and the share of migrated reads grows gradually, based on error and latency metrics.
Only after reconciling counters and checksums does the old table go into reserve mode rather than being deleted.
Indexes and capacity mode are simpler: a global index can be added and removed on a live table, but it changes the write profile, so it is introduced in advance and tested on a copy of traffic.
Capacity mode switches without recreation — it is only important that it matches the nature of the load, otherwise peaks go into request throttling. A local index cannot be changed: only via a new table, and this is another reason to do the migration once.
Rollback at any step means returning reads to the old table. It remains live until confirmed stability, so the rollback window is measured in minutes, not in restoring from a backup.
Doubts and Limitations: Point by Point
Technical doubts usually surface already at the evaluation stage — and that's normal: they are cheaper to resolve before migration than after. Below are the questions customers ask most often when discussing AWS DynamoDB, and our answer to each.
Separately about hypothesis testing: we run access keys, schema headroom, and behavior under load on a stand with a production query profile, not on empty tables. This way migration risks become visible before they turn into an incident.
How difficult is it to change the capacity mode later?
Switching between provisioned throughput and on-demand mode is performed without downtime and without recreating the table — only the capacity allocation method and autoscaling logic change. Switching back is also supported, so the chosen mode does not lock you into one option.
What are the dangers of full table scans?
A scan reads the entire data set: response time grows linearly with volume, and capacity consumption hits neighboring queries. We minimize such accesses — key-based access, secondary indexes for specific query patterns, restricting scan permissions for service accounts.
Will the key schema be sufficient for growth?
We design the schema from access patterns, not from object structure: we check the distribution of records across partitions so that volume growth does not create "hot" keys and does not hit one segment.
We allow headroom for an order of magnitude more than the current volume — rebuilding keys on a live table requires a separate data migration.
Who is responsible for access separation?
We configure roles and access policies: separate accounts for services and engineers, least-privilege permissions, logging of data access. Any permission change goes through approval and is recorded in configuration — access does not grow spontaneously over time.
Let's Discuss the Task: What We Need from You and How We'll Start
To come back with an estimate and plan rather than generalities, we need inputs on load and clear data access scenarios.
The more accurately you describe the profile, the fewer assumptions we will build into the key schema and capacity mode of DynamoDB tables — reworking the model after launch costs more than analyzing it before.
What is useful to send or tell us on a call:
- load profile: average and peak request intensity, daily and weekly spikes, expected volume growth;
- read and write scenarios: which queries predominate, by which keys searches are made, whether range queries and aggregates are needed;
- entity examples or current schema: fields, relationships, record sizes, and retention periods;
- fault tolerance requirements: acceptable downtime, whether backups and point-in-time recovery are needed;
- access and environments: how many environments, who and how accesses the database, whether there are restrictions on role separation;
- constraints: change windows, approval procedure, whether migration of existing data is needed.
If some inputs are still unknown — that's normal, we'll capture them during the review. Then we fix the key and index schema, migration and rollback procedure, scope of work, and acceptance criteria.
Describe the task — we'll come back with an estimate and plan and propose a start date for the work.