Synthetic data platform development: where the test bench breaks
The test bench breaks not in training, but in the data beneath it. The team assembles a sample from whatever is at hand: production stream exports for a couple of weeks, a copy of some records, manual examples from analysts.
The model converges, metrics on the holdout look even, but on rare scenarios — long sessions, interruptions, atypical amounts — quality drops severalfold, and this surfaces only during manual error analysis.
The second fork is data privacy during development. A full copy of production records cannot be taken outside the perimeter, and masking breaks exactly what the model must learn: relationships between entities, event chronology, co-occurrence of features.
The resulting dataset formally resembles production, but its structure is scattered — hypotheses are tested on it and get a false "doesn't work," although the cause is in the data.
The lack of real data completes the picture: rare classes — failures, disputed transactions, anomalies — occur in single digits per tens of thousands of records.
It is impossible to assemble a test set for the model with a meaningful number of examples from them, so precisely what the system is built for does not make it into validation.
Synthetic data platform development changes the source: the generator reproduces dependencies and frequencies from statistics and rules rather than copying rows.
Synthetic data for testing gains controllable complexity — a specified share of rare classes, noise, missing values, distribution shift between training and validation.
Every release passes quality contracts (Great Expectations) and measurement of the gap with the real sample — next we'll break down how this works.
What changes after implementation: speed, privacy, control
The first effect is visible at project entry: a training set is assembled not over months of exports and approvals, but in a matter of days.
Dataset generation follows a description of the schema, rules, and distributions, so rare scenarios — payment failure, session interruption, duplicate request — enter the sample in a specified proportion, not "as luck would have it."
The second is removing dependence on the production environment. Previously, every iteration ran into a request for access to production data, review, and waiting for an export window.
Now the dataset is reproduced from configuration: the same set of fields, the same volume, the same statistics — including periods for which real data simply does not exist.
Privacy stops being a compromise. Record anonymization is performed at the generation stage, not on top of a finished file: the dataset contains no direct identifiers or traces of real people, but relationships between fields and distributions are preserved.
Sample representativeness is checked automatically — a report by slices shows in advance where the model will have too few examples.
Model training speed grows not from added hardware, but from input quality.
When the error class the model will see at input is specified in advance — class balance is set, noise and duplicates are removed — training converges in fewer iterations, and the metric on the validation set stops changing from run to run.
Generation approaches: rules, statistics, trainable models
The choice of generation family determines which checks we can guarantee, not the "beauty" of the data. Rules and templates provide a deterministic result: every field is explainable, relationships between tables are not broken.
Statistical models reproduce feature distributions but are not responsible for the meaning of a record. Trainable generative models provide plausibility but require a data corpus and a quality control loop.
In practice, these families do not compete but join: rules define the frame and constraints — referential integrity, permissible values, formats — a trainable layer fills content fields, a statistical one monitors joint distributions and the share of rare classes.
| Approach | When suitable | Limitations |
|---|---|---|
| Rules and templates | Reference data, identifiers, transactions with rigid logic; determinism and traceability of every field are needed | Weak diversity, growing maintenance cost with frequent schema changes |
| Statistical models (Markov chains, Bayesian networks) | Numerical and categorical features where distributions and correlations matter | Do not hold long dependencies and rare events; sensitive to sample size |
| Trainable generative models | Texts, images, free-form fields, complex nonlinear dependencies | Require training data, quality filters, and explanation of the result |
We assess suitability along three axes: the share of records that passed validation, resilience to schema changes, and labor costs for maintenance.
A hybrid of rules and a trainable layer gives the best balance: the frame remains verifiable, while variability comes from the model.
Phased implementation: dataset audit, schema, validation, handover
A synthetic data platform delivers a predictable result only if the generator relies on measured properties of the data, not assumptions.
Therefore, data platform implementation proceeds from data source audit to platform acceptance: each stage is closed with an artifact that can be verified independently of us.
- Data source audit — we profile samples: distributions, cardinalities, missing value shares, relationships between fields. We record which dependencies the generator must reproduce and which we consider noise.
- Schema and generation rules — we describe tables, types, integrity constraints, category shares. The generator maintains joint distributions, not just individual fields.
- Privacy control — we check that synthetic data does not repeat real records and cannot be reconstructed from indirect attributes.
- Quality validation — we compare the distributions of synthetic and original data by metrics, train a control model on synthetic data and validate it on real data.
- Platform acceptance — a checklist run on your scenarios; thresholds for quality and generation stability are agreed in advance.
- Deployment in your environment and handover — we deploy in your infrastructure or an isolated environment, set up scheduled runs, rotation, and logging, provide documentation and procedures, and run a new scenario together with your team.
Configurations, schemas, and procedures: what we hand over to your environment
A platform without documentation quickly turns into a black box: generation works, but only the person who configured it knows why the distributions look exactly this way and how to connect a new source.
Therefore, handover includes not only the working environment but also a set of artifacts your team can use to develop it independently.
- Generation profiles — saved configurations for each scenario: rules for relationships between tables, missing value shares, masking of sensitive fields, fixed seed for reproducibility.
- Relationship schemas — which entities depend on each other, where referential integrity is mandatory, which values are considered derived and are not generated directly.
- Integration schemas — connection points of sources and sinks, exchange formats, export order, behavior when an adjacent system is unavailable.
- Data handling procedures — what enters the environment, which fields do not leave your perimeter, copy retention periods, the procedure for approving new datasets.
- Data access rights — roles on the principle of least privilege, separation of generation operator, administrator, and reader, action logging.
- Platform documentation — architecture description, operating instructions, the procedure for updating profiles, and standard failure recovery scenarios.
- Acceptance checks — control datasets and metrics by which you verify data quality after every configuration change.
We hand over configurations and schemas in a version control system so that any profile change is visible in history and can be rolled back with one command. After that, the environment lives on your side: updating profiles and adding sources does not require our involvement.
Case: a test bench on anonymized datasets instead of a production copy
What was before implementation
The test environment relied on a production copy: a database snapshot once a quarter, manual masking of personal data, adding missing records to tables.
Preparing one dataset took 3-4 weeks and constantly occupied two engineers; the share of manual labeling reached 40% of records.
At the same time, about a third of scenarios from analysts' requirements were not covered — checks failed in every second release, and some defects went further down the pipeline.
What was changed
The data schema was described as a contract: entities, relationships, value distributions, admissibility rules. The platform assembles datasets from this contract — real records are replaced with model-generated ones, referential integrity and constraints are preserved.
Datasets are versioned and reproduced by run number; rare edge cases are defined by rules, not rows in tables. Generation and validation runs are scheduled in Airflow: a dataset is rebuilt for a changed schema in minutes.
What became
| Indicator | Before | After |
|---|---|---|
| Dataset preparation time | 3-4 weeks | 1 business day |
| Share of manual labeling | up to 40% of records | less than 5% |
| Check failures due to insufficient scenarios | every second release | 1 out of 20 releases |
| Dataset reproducibility | not guaranteed | identical by run number |
After two quarters, a production copy in the test environment is no longer needed: personal data does not enter the development environment at all, and scenario coverage grows without increasing manual work.
Is a platform needed if anonymized datasets are already assembled?
An anonymized archive answers the question "what has already happened." If the model is trained on historical records, labels are preserved, and the schema is stable, a separate platform is not needed: an export, integrity check, and field description are enough.
A separate environment becomes mandatory when events are required that are scarce or absent in the archive — rare failure classes, atypical feature combinations, edge modes that almost never occur in production operation.
The second limiter is the density of rare cases. In a live stream, the share of an anomaly may be fractions of a percent, and for stable training it is physically insufficient: the model simply does not see the class a sufficient number of times.
A synthetic data platform addresses this by generating to specified distributions and explicitly controlling class balance — instead of waiting for more years of observations.
The third is reproducibility. An archive does not describe how it was obtained: which filters were applied, by what rules identifiers were removed, what was excluded. When model quality drops, it is unclear whether the issue is in the data or in training.
The platform stores the dataset recipe — generation parameters, generator state, feature composition, and consistency rules — and allows recreating exactly the same dataset.
The choice between a ready-made solution and an in-house platform comes down to dataset scaling. A ready-made tool covers typical structures while schemas are standard and there are one or two datasets.
An in-house platform is needed when there are dozens of datasets, each with its own relationships between tables, its own constraints, and its own requirements for the share of rare classes.
Then generation, versioning, and delivery of datasets become a shared environment rather than manual work for each request.
When choosing a contractor, check not the demo, but how they answer three questions: how they will prove that synthetic data does not degrade model quality on real data, how they will transfer their work into your environment, and what will happen when the source schema changes.
The answers to these questions show whether you will get a working environment or a one-off dataset.
Technical concerns: representativeness, drift, acceptance
We check dataset representativeness by comparing distributions, not by visual inspection: for each feature, we calculate category shares, quantiles, and pairwise correlations separately for the real and synthetic samples.
A deviation above the threshold fixed in the acceptance criteria sends the dataset for reassembly — otherwise rare but significant classes are averaged out and fall out of the training sample.
We monitor data drift on a schedule: we compare the current source profile with the reference one and receive a notification when the schema changes or the distribution of a key field shifts.
We describe validation rules declaratively and store them next to the generator — they are versioned and rerun on every run, so a source change does not go unnoticed.
We agree acceptance criteria before generation: the list of checks, thresholds, the test slice that did not participate in training, and the procedure for handling deviations. Below are answers to objections that engineers most often raise during acceptance.
How is similarity to real data proven?
We run automated checks on Great Expectations — value ranges, key uniqueness, consistency of relationships between tables, plus statistical metrics of distribution proximity. The output is a report for each run: you can see which rule fired, at what threshold, and in which field.
What to do if sources have changed?
We rebuild the source profile, update the rules, and regenerate only the affected tables — the rest remains valid. Previous dataset versions are preserved, so returning to the previous state takes minutes.
Is one reference slice enough for acceptance?
No. We check on several slices — by periods, segments, and intake channels. An overall metric hides local bias: a dataset may look correct as a whole while distorting the behavior of one segment on which the model is precisely trained.
Let's discuss the task: we'll show the approach and come back with an estimate
To make the conversation concrete, the task outline is enough: what data types need to be synthesized, what statistics on real records are available, where the platform will be deployed, and what access and anonymization restrictions are already in place. Describe this in free form — our engineers will ask clarifying questions themselves.
We will come back not with general wording, but with a point-by-point breakdown:
- scope of work by stages — from profiling the real dataset to validating synthetic data;
- timeline estimate with the assumptions it rests on: data volume, number of sources, validation requirements;
- implementation plan: pilot environment, then expansion to the customer's remaining systems;
- scheme for generation, storage, and delivery of datasets — with markers showing where resources are needed on your side;
- acceptance metrics: how we prove that synthetic data preserves distributions and does not leak on privacy;
- requirements for access, role separation, and logging of operations;
- list of risks and decision points that should be fixed before work starts.
If some data for answers is not yet available, say so, and we will propose a way to collect statistics without disclosing the records themselves. Describe the task — we will come back with a timeline estimate, scope of work, and implementation plan.