What breaks without synthetic test data
A test environment is usually populated with an export from the production database — a snapshot of a couple thousand records taken long ago and unchanged since.
Checks on it are green, while in prod a scenario that was not in the snapshot fails: an order with two currencies, a cancellation after a partial refund, a merge of two accounts.
The shortage of test datasets hits rare states hardest.
Their share in real data is fractions of a percent, they do not make it into the sample, and flaky checks for years see neither a race during concurrent write-off, nor a timezone divergence, nor half of the branches for handling external service failures.
A separate problem is real data in tests. Copies of production tables go to test environments together with personal data, access permissions, and payment history, and the circle of people with access there is wider than in prod. Such copies cannot be given to a contractor and are difficult to reproduce locally.
Masking on top of a copy helps only partially: relationships between tables break, reference data and foreign keys stop matching. The test either fails for a reason unrelated to the code or passes on inconsistent data — and both situations undermine trust in the run.
As a result, checks confirm not that the system works, but that it works on one narrow snapshot. We explain how this is addressed with synthetic test data: generation of datasets with specified properties, controlled rarity of states, and reproducibility from run to run.
What generation delivers: measurable effects
Environment preparation stops being a waiting game. Previously, a test environment was assembled from a copy of the production database: export, wiping personal fields, manual correction of referential integrity — and all over again after every schema change.
The generator builds a dataset from the schema description and relationship rules, so assembly is reduced from days to tens of minutes and is repeated during migration without manual steps.
Rare scenarios stop being rare. In production data, cancellations mid-process, zero and negative values, duplicates, and boundary field lengths occur in tenths of a percent — tests for them are written almost blind.
By setting the shares directly in the rules, we get a dataset where such combinations account for a noticeable portion of records, and the defect is caught before release, not by the first client.
The dataset is reproducible. The rules and a fixed generator seed produce the same result on any machine and in any branch: a found bug is reproduced by any engineer without sending exports.
A regression comparison of two runs shows what changed in the code, not in the data — the test result stops depending on who updated the test database and when.
Data leakage ceases to be a class of incidents. A copy of the production database in a test environment or on a contractor's machine is the most common path out, and it is also the least noticeable: access formally exists, and no one investigates the event.
A synthetic dataset contains no personal data by design, so it can be used for testing without additional approvals, and masking remains only where the environment still runs on copies.
Comparison of approaches to synthetic data
An approach is chosen not by the “quality” of the data as such, but by the properties tests need: reproducibility of a specific case, coverage of rare combinations, schema conformance, and referential integrity. Below are three options among which teams choose in practice.
The difference shows up at scale: rules stop holding correlations between fields, while a generative model drifts from the real distribution.
Therefore, we look not at a sampled fragment, but at how the approach behaves at the full volume of the stand — the simplest rules layer on Faker provides reproducibility but not joint distributions.
| Option | When it fits | Limitations |
|---|---|---|
| Templates and rules | Bulk schema-valid records: load runs, migration checks, populating stands | Fields are generated independently, correlations between them are written manually; rare combinations almost never occur |
| Generative models | Realistic distributions and anomalies: model training, tests for rare scenarios | A representative sample is needed; risk of transferring personal data; the result is unstable, and an individual case is difficult to reproduce |
| Combined approach | Both scale and realism: rules hold the frame, keys, and relationships, the model adds variability | A contract between layers and validation of each step are needed, otherwise a failure of one layer is masked by another |
The combination wins more often because it separates responsibilities: rules are responsible for referential integrity and scenario reproducibility, the model — for plausibility and rare cases.
Acceptance is the same for all options: running the synthetic data through the same schema and invariant checks as production data — a dataset that fails them does not go into tests.
How generation is built: stages of engineering work
The work proceeds in iterations: first we capture the actual data picture, then we build the generator, then we prove that the datasets are suitable for checks.
The reverse order produces plausible but useless data — it does not reproduce relationships between entities and rare combinations, which are what production environments most often fail on.
Implementation stages
- Source inventory. We record systems, export formats, volumes, locations where personal data is stored, and rules for its de-identification.
- Data and relationship profiling. We calculate distributions, field cardinality, missing-value rates, correlations, and referential integrity rules — without this, the generator stamps out “orders without customers.”
- Generation rule setup. We define types and ranges, reference data, business invariants such as totals matching, and separate scenarios for rare and boundary combinations.
- Generator assembly. We configure modes: a structure snapshot, a dataset of the required volume, reproducibility by a fixed seed — a failed check is investigated on the same dataset.
- Dataset acceptance. We compare metrics with the source and run checks: they must fail on planted defects and not trigger on clean data.
- Handover of the update procedure. We describe how to change rules when the schema changes, where configurations are stored, and how datasets are versioned.
This order makes generation repeatable: a change to the source schema does not silently break the checks.
What we hand over: artifacts and procedures
Datasets without accompanying documentation last exactly until the first schema change: the author leaves, and no one can explain where the value in a disputed field came from.
Therefore, together with the data itself, we hand over everything needed to reproduce and evolve it without us.
- Generation configurations: distribution rules, field constraints, value reference lists, and initial parameters under which the dataset is assembled exactly the same way.
- Table relationship schema: a map of foreign keys and generation order in which child records receive consistent parent records.
- Access rules: which teams and services are allowed to read the datasets, who has the right to run generation and change its parameters.
- Dataset documentation: entity composition, volume, distribution of key fields, and a list of scenarios the dataset covers.
- Update and re-run procedure: what triggers a rebuild, which checks are run, and by what criterion the dataset is accepted for use.
- Integrity control scripts: checking references, duplicates, and values exceeding allowed boundaries — performed before the data is issued to teams.
- Coverage report: which logic branches and failure classes are covered by generated scenarios, and which are deliberately left to real data.
A few weeks after handover, you run the rebuild for a new schema yourself — without coming back to us for every change.
Case study: reduced manual environment preparation
The client prepared test datasets manually: export from the production database, cleaning personal data, adding missing relationships.
One dataset took about five days, and failure scenarios — partial records, duplicates, transaction breaks — were not covered at all, because such data simply did not occur in the database.
We built a generator that reads the PostgreSQL schema and assembles a consistent dataset: foreign keys, constraints, and insertion order are reproduced just as in a real database.
Personal values are replaced with plausible ones, and the generator's initial state is fixed in the configuration — the same dataset is reproduced on re-run.
Controllability turned out to be more important than volume: the generation profile is described declaratively — anomaly rates, distributions by category, rare states.
A new scenario is added by editing the profile rather than rewriting the generator, and the dataset is assembled for a specific check.
What changed
Dataset assembly takes minutes instead of five days, the share of covered boundary scenarios grew from 40% to 95%, and manual data edits are gone from the release process. Engineers no longer wait for an environment: a clean database comes up with a single command.
A separate effect is defect reproducibility. A failure on a synthetic dataset is reproduced by any engineer with the same parameters, so the cause is found in hours rather than days of “it doesn't reproduce for me.”
What to do if de-identification is already implemented
If de-identification is already implemented, it does not need to be redone — it is not an alternative to generation, but a different tool.
Masking preserves structure and relationships between fields: values are replaced according to dictionaries, but the same customer is still linked to their orders and history.
Therefore, a copy is well suited for debugging and regression and poorly suited for testing behavior on rare classes.
Masking collapses the distribution. A rare code, a one-off combination of conditions, an anomalous event order after replacement lose frequency or look ordinary.
Deterministic replacement does not allow such cases to be multiplied: the copy has exactly as many as there were in the source, and the sample can no longer be increased — the dataset is fixed.
The practical step is to embed generation alongside, using the agreed de-identification schema as a specification.
Replacement dictionaries, relationship rules, and permissible field ranges become input rules for the synthetic test data generator: it inherits the structure of the copy but is not limited by its volume and produces the required number of runs, including rare combinations.
Migration is gradual. First, generation covers what the copy does not provide: volume for load runs, boundary cases, rare classes. Reference datasets for regression remain on de-identified data, and the process does not break — only the source of some datasets changes.
Control is mandatory: we compare distribution profiles and referential integrity rules between the de-identified and generated datasets so that impossible combinations do not appear.
The division is simple: de-identification answers the question “did data leak,” generation answers the question “is there enough data and are rare classes covered.”
Technical concerns: leaks and reproducibility
Engineers ask the same questions before starting: will a real export end up in the dataset, how is its quality confirmed, and what happens when the schema changes. Below are answers without generic phrases.
Can real data end up in synthetic data?
The generator works from a description of the schema, distributions, and relationship rules, not from an export from the production environment.
We take a real snapshot only for distribution comparison — de-identified and at the analysis stage; it does not go into the finished dataset.
How can you make sure the dataset is high-quality?
We run the dataset through the same checks as production data: schema constraints, key uniqueness, referential integrity, category shares by slices. Separately, we compare distributions with the reference so that new records do not shift them unnoticed.
Is the dataset reproducible on another machine?
Yes: a configuration with a fixed generator seed produces the same dataset on every run. If the seed, rules, or schema change, it is already a different dataset, and the passport shows what it was assembled from.
What happens after a schema change?
Regeneration follows the rules: new fields are described, outdated ones are marked, and ready datasets remain unchanged. Tests tied to the previous dataset continue to pass until you migrate them to the new one.
Will the dataset withstand load checks?
Volume is set separately from plausibility: the generator scales to the required number of records while preserving relationships between entities. This way, system behavior is tested at volumes that do not exist in the production environment.
Let's discuss the task and come back with an estimate and plan
To make the estimate based on facts rather than guesses, it is enough to describe the task in free form: which systems provide the data, in what form records arrive, where the boundaries of sensitive fields lie, and by what metrics you will understand that the dataset is suitable for work. The more specific the inputs, the more accurately we will estimate the scope and timeline.
On our side, we return:
- source analysis — which tables and relationships need to be covered, where there is not enough data for a realistic distribution;
- generator schema: record relationship rules, distribution by fields, handling of rare classes and boundary cases;
- sensitive data handling policy — masking, replacement with synthetic data, reversibility check;
- a stage-by-stage work plan with result checkpoints at each stage;
- a pilot dataset on one area — so that quality is checked on data, not on a presentation;
- acceptance criteria: share of passing checks, model stability on synthetic data, match of distributions with real ones;
- handover composition: configurations, relationship schemas, regeneration procedure when the source changes.
Describe the task and current constraints — we will come back with an estimate and work plan, and we will assemble a pilot dataset on your data so that the approach is confirmed before the main stage begins.