Schema evolution in NoSQL: where production breaks
In document databases, the schema lives in the records themselves, not in a separate table: migration does not block writes — and this is the main reason why schema evolution in NoSQL is postponed until the first failure.
The code has been updated, collection validation rules enabled, but the data remains in two or three forms at once. In MongoDB, it is enough to add a required field to the validation rule — and some existing records will stop passing validation on update.
The first point of failure is a document read crash. New code expects an attribute that is absent in the old record: the object does not pass validation, and the handler returns an error.
While such records are rare, the failure looks random — one request out of a hundred fails, and it cannot be reproduced on fresh data.
The second is structural desynchronization on required fields. The rule validates the entire record, while a targeted update writes a partial set: some documents remain without the new attribute, and the rule starts rejecting the service's own records.
The third is batch processing on mixed record versions. A job reads a sample, stumbles on the first record in the old format, and either fails entirely or writes half the set in the new form — the set remains in an intermediate state, and it has to be sorted out manually.
We look at how this is addressed: record versioning, reading that accounts for both forms, and backfilling without stopping the service.
What managed record structure changes provide
Compatibility is checked before application
A record structure change is analyzed for backward compatibility across versions: an old application version simply does not notice the addition of a field with a default value, while deletion or renaming is split into two phases with an intermediate state.
Only changes that are readable by both code versions reach the production environment, so the share of changes rolled back due to incompatibility drops to zero.
Rolling back a change takes minutes instead of hours
The structure version is described separately from the documents themselves, and returning to the previous one is a descriptor switch, not an export with a reload of the collection.
On datasets of tens of millions of records, rollback time is reduced from hours to minutes, and it can be performed during a working window.
Read failures become a managed class
Previously, a structure mismatch produced an unhandled exception for the entire request.
Now a field missing from an old record is supplied by a read transformation, and the share of requests failing due to version mismatch drops to zero — only isolated cases remain, visible in metrics.
Data is rewritten lazily, on access
Mass migration is not needed: the transformation is applied to the document at read time, and it is saved in the new structure on the next update. The volume of rewritten data is reduced to the share of actually used records, and long-running locks on the collection do not occur.
Manual edits become governed, and releases speed up
Changes go through review, are recorded in version history, and are deployed in the same way as code. The share of edits that bypass the process drops to isolated cases, and a coordinated release of a structure change takes days instead of weeks.
Approaches to schema evolution: comparison of options
The schema in a document database is not described in advance, so the question is not whether validation is needed, but when it takes effect.
Validation on read, strict validation on write, and an explicit record version provide a different balance between write availability and read predictability.
| Option | When it fits | What limits it |
|---|---|---|
| Validation on read | Heterogeneous sources where stopping writes is unacceptable: the document is written as is and parsed on the reader side | The error appears later, and compatibility logic spreads across the code of all consumers |
| Strict validation on write | A single data owner, high cost of an incorrect document — the failure is visible to the author immediately | Schema expansion requires simultaneous updating of all writing services, and old versions begin to be rejected |
| Explicit record version | Long-lived data and exports: you need to read documents written months earlier | It requires transformation rules between versions and discipline in keeping them up to date |
| Hybrid validation | The core fields are validated strictly on write, optional extensions — on read | It requires an explicit boundary between the required and extensible parts of the document |
In practice, the choice is determined not by development convenience, but by where we are willing to pay for an error.
Hybrid mode covers most scenarios: required fields prevent writing a knowingly broken document, while the free part allows adding attributes without releasing all services at once.
If data is read for longer than the code lives, the record version becomes mandatory — otherwise restoring an old export turns into archaeology.
Implementation stages: from audit to handover
The data structure in NoSQL cannot be changed in a single release: documents of previous forms live in the same collections for months, and the service must read both forms all that time.
That is why we conduct the transition in sequential steps, where each state is compatible with the previous one.
- Inventory of record versions. We count what share of documents corresponds to which form: where a field type has diverged, where fields from rolled-back releases remain, where nesting differs. We get a map of the actual state of the data, not assumptions based on code.
- Compatibility scheme and structure change plan. We divide changes into backward-compatible and breaking: the service survives an optional field immediately, while renaming or type change — only in two phases, through a period of reading both variants.
- Setting up validation rules. We enable the validator in soft mode: it writes mismatches to the log and does not block writes. After a few days, it is clear what exactly violates the new rules, and code changes are made based on facts.
- Trial run on a data copy. We deploy a snapshot of the working database in an isolated environment and run the transformation together with rollback: pass time, share of records not transformed automatically, service behavior on mixed data.
- Gradual enablement of rules. We move validation from soft to strict mode collection by collection, starting with low-load ones. Control — the share of records passing validation and the frequency of read errors at each step.
- Handover of the process. We deliver the document: how to add a field, what is considered a breaking change, how a two-phase transition is formalized, who initiates rollback, what checks are in the build. After that, the team changes the schema on its own.
Deliverables: rules, processes, documentation
Deliverables are not a presentation or a folder of screenshots, but working files in the repository and the team wiki, which the next engineer can use to continue the work without verbal explanations.
Each document answers a specific question: what is currently in the data structure, what can be changed safely, and how to act if a change goes wrong.
- Description of structure versions: which fields appear, which stop being populated, what remains readable for records of previous versions, and until what point they need to be supported.
- Version compatibility rules: what can be added without stopping services, and which changes require parallel writing of two versions and an intermediate read layer.
- Record processing scheme: the document path from write to read — how the reader parses different versions in one collection and where a new parsing branch is added.
- Data access policies: who changes the structure and through which role, where a second approver is needed, how access to sensitive fields and exports is restricted.
- Change release process: deployment order, allowed windows, a set of automated checks on a data copy, and the criterion at which a release is stopped.
- Rollback process: pre-described steps for returning to the previous structure version and, separately, what to do with records that have already been written in the new format.
- Structure documentation: an up-to-date map of collections, field purposes, relationships between entities, and the date of the last review, so it does not have to be reconstructed from code.
- A set of verification scenarios on a data copy: typical and edge cases on which a change is tested before production.
All of this is handed over together with the setup, rather than remaining with us: the customer's team can make changes to the structure itself without breaking compatibility.
Case study: structure change without service downtime
What was the situation
The order service stored documents in a NoSQL MongoDB store without a schema version field: the application read them directly and expected exactly the set of fields that had once been written.
As soon as a nested delivery address block was added to the structure, some handlers began crashing during parsing — the share of document read failures rose to 4–6% of requests.
Record versions were migrated with a one-off script that stopped new writes: a structure change took 2–3 weeks, and on each iteration dozens of documents were edited manually.
What was changed
We added a version field to every document, and a transformation layer on read: the record is stored in its original form, the application brings it to the current contract, and it does not depend on when the document was created.
We fixed compatibility rules: new fields must be optional only, renaming — through a duplicate field for the transition period, deletion — only after no records of the old version remain.
The share of old-version records in the first month fell from 100% to less than 1% — a background job rewrites documents as they are accessed, without pauses in service. Read failures on new fields disappeared because the read contract is tested on both schema versions.
The time to release a structure change dropped from two to three weeks to two days, manual edits dropped to zero, and rolling the application back to the previous version no longer requires data migration.
What to do if the store is already in production with different versions
The spread has already accumulated: one collection contains documents of different generations — with renamed fields, without fields that appeared later, with changed types.
You cannot rewrite everything at once: writes to the store do not stop, and any run will encounter a document that has not yet been brought to the required form.
So the first step is to introduce a schema version marker directly into the document — it is used to decide how to interpret the record.
Reading mixed records is handled by a data normalization layer: missing fields get default values, renamed ones are expanded to the current name, and types are brought to the expected ones. Application logic remains in a single variant and knows nothing about the spread of versions.
Next is gradual alignment: a background pass goes through the collection with a cursor in small batches, brings documents to the current form, and marks them with a new version.
It is resumable after interruption, does not hold long locks, and does not compete with the production workload. We keep the share of old-version documents as a metric — it shows when the pass is actually complete.
The main thing is not to stop writes. A new field is added additively, the old name is written duplicatively for the transition period, and code that has not yet been updated continues to produce valid documents.
Removing the old branch is the last step, when the counter of previous-version records reaches zero.
Validation, rollbacks, and compatibility: common questions
Strict record validation hits not new but old documents: a rule with a required field cuts off everything written before it.
That is why fields are added as optional with an explicit default value, and requiredness is enabled when the counter of records without the field — taken from the store itself — drops to zero.
Rolling back a structure change is not a binary operation. While the application writes in two formats, it is more reliable to return the previous set of rules: it accepts both old and new records, and partially updated data does not block reads.
The boundary of responsibility is usually as follows: the store validates structure and types, the application — business rules and field consistency.
What happens to old documents in the database?
We do not rewrite anything in a single pass: migration proceeds in batches, and reads substitute the default value until the document is updated. Migration lag does not affect service operation.
Can a structure change be rolled back if data has already been partially written in the new format?
Yes, if the schema is extensible: old rules ignore unknown fields, and new ones do not require their mandatory presence. A full rollback is not needed — it is enough to return the previous set of rules.
Where does the store's responsibility end and the application's responsibility begin?
The store holds structure, types, and allowed value ranges. Relationships between entities, permissions to change, and computed fields are the application's task: rules of that level cannot be expressed inside the database and should not be.
How can version compatibility be verified before rollout?
We run reads and writes with the new code on a copy of the working data set and compare validation failure counters. If the increase in failures is zero, we enable the rules, keeping the previous set ready for return.
Let's discuss the task and return with an estimate and plan
To ensure the data work plan is based on facts, describe the current state: how many collections there are and which document variants coexist in them, whether there is already a version field, how readers react to a missing field, which application releases run simultaneously during deployment.
Add constraints — a maintenance window, consistency requirements, data volume, and what cannot be changed.
In return, we send a substantive document, not a set of general practices:
- data work plan: inventory of schema variants by collection, the share of documents of each kind, and places where the mismatch already breaks reads;
- composition of stages: a version field in the document, read rules for several versions, deployment according to the "readers before writers" scheme, background enrichment of old records;
- control points: the share of old-version documents, the parse failure counter, an alert on an unknown version, tests for reading each supported kind;
- backward compatibility policy: changing a field type or name — only through a period of dual writes, deletion — after the old-version counter reaches zero;
- effort estimate by stage: what we cover with our own resources, and where decisions are needed from your side;
- rollback plan: which steps are reversible and at what stage the point of return is.
Describe the task — current collections, constraints, what you have already tried. We will come back with an estimate, stage composition, and control points by which you will accept the work.