AI data profiling: where quality breaks the model
A typical failure looks like this: training data was collected from a sales system, billing, and a marketing export; the model showed steady metrics on a holdout sample — and in production it produced nonsense. The cause is almost never the model architecture.
The dataset arrived with duplicate customers from two accounting systems, missing values in the fields used to compute a feature, and anomalies in records: negative age, a closing date earlier than the opening date, test employee accounts.
The model honestly learned this noise. Then begins the most expensive part: engineers look for a defect in the code, analysts — in the features, and the business unit lead loses trust in the project itself.
Retraining on the same data gives the same result, because the root cause is data quality, not model settings.
AI data profiling moves this check to the beginning of the work. The agent goes through columns, calculates distributions, missing value rates, key intersections, looks for anomalies in records and flags rows that will spoil training.
Validation rules in Great Expectations or custom checks are formalized as a versioned set — the profile becomes a project artifact, not a one-off script in someone's folder.
Practical effect: analysis takes hours instead of weeks, defects are caught before training, not after rollout, and the argument "is the model bad or the data" is closed by a column report.
Below we break down how this is handled in a project — from source inventory to control of new arrivals.
What changes in numbers after quality assessment
Profiling runs the entire dataset before training: for each field, it calculates missing value rate, value distribution, duplicates, outliers and format violations, and separately — type discrepancies between sources.
Already at this step it is clear which features are unsuitable, and they are discarded before labeling, not after the model produced an inexplicable result at acceptance.
The effect is measurable at three points. Data preparation time is reduced severalfold: manual table reconciliation is replaced by a column report that is rebuilt on every run.
The share of defects in records drops from double-digit percentages to single digits — mass defects such as empty categorical fields, duplicates and shifted date formats are cleaned in batches, not one record at a time.
The third point is the classes of failures that disappear entirely.
Type desynchronization between sources, duplicates of the same entity under different keys, time shifts and garbage values that got into features no longer reach the model: they are caught by record quality control at the input.
Rules are described declaratively, for example in Great Expectations, and checked automatically on every load.
Then the profile is fixed as a contract: a new batch of data is compared with the reference slice, deviations are highlighted before entering training.
Data suitability ceases to be a matter of faith — it is confirmed by a set of checks, not by a feeling that "everything seems fine".
Analysis modes: comparison of approaches and limitations
Choosing a mode is not a matter of taste, but a consequence of where the data resides and how quickly a verdict is needed.
Batch data validation takes the entire export and calculates statistics across the whole dataset: the picture is complete, but analysis time grows with volume.
Streaming data validation looks at records as they arrive — quality metrics appear within minutes, but the retrospective remains out of view.
In practice, the scheme is almost always hybrid: the stream maintains control of freshness and schema drift, while a periodic full pass refines distributions and finds rare value classes that the stream did not have time to notice.
| Option | When it fits | Limitations |
|---|---|---|
| Batch over full export | initial dataset assessment, audit before migration or loading into storage | analysis time grows with volume; the picture is fixed at the time of export |
| Batch over stratified sample | quick verdict on tens of millions of rows | rare values and tail anomalies may not make it into the sample |
| Streaming at pipeline input | real-time quality control and early notification of schema failure are needed | no retrospective, comparison is only against the accumulated window |
| Hybrid: stream plus periodic full pass | production pipelines where both freshness and completeness matter | harder to configure: two loops need to be coordinated by rules |
A limitation common to all modes: the profile describes statistics, not the meaning of a field. What a value means and which ranges are acceptable is confirmed by the data owner — without this step, the analysis remains a set of metrics without conclusions.
How implementation proceeds: stages of engineering work
Implementation of profiling proceeds in waves: first we capture facts about sources, then fix rules, and only then enable control in continuous mode.
The order is fundamental — validation rules written before the source audit catch not what actually breaks the data, and the team quickly stops trusting them.
- Source audit — we go through systems, reference data and exports: formats, update frequency, owner, how data gets into storage. The output is a map of where a source is considered authoritative and where data is collected manually.
- Slice profiling — for each column we calculate fill rate, number of unique values, lengths, distributions, date formats. This shows the actual semantics of fields, not the ones stated in the description.
- Validation rule scheme — a set of checks grows out of the profile: requiredness, ranges, referential integrity, consistency of fields with each other. For each rule we immediately define the reaction: stop loading, quarantine the record or flag it.
- Anomaly threshold tuning — we select the threshold on historical data. Too strict gives a stream of false positives, too soft lets degradation go unnoticed; a rule is considered working when it is known how many triggers per week it gives on a normal stream.
- Check on a control sample — we run the rules on a slice with known defects and compare the result: what was caught, what was missed, where the noise is. After this, two or three rules are usually rewritten.
- Handover of the regulation — we provide the description of rules, the procedure for reviewing triggers and areas of responsibility. Then the customer's team changes thresholds and adds checks independently.
What we hand over: configurations, schemas, regulations
We deliver not a report with conclusions, but a working kit with which the customer's team continues to live without us.
Each artifact is versioned, stored in a shared repository next to the code and reviewed during handover — a new engineer joins the project by documents, not by chat.
- Source schema: list of supplier systems, owners on the customer side, data arrival mode, failure points and actions in case of unavailability.
- Data documentation: description of datasets and fields, units of measurement, acceptable ranges, history of structural changes and reasons for those changes.
- Feature dictionary: for each field, meaning, type, origin, normalization rules and a list of synonyms by which the field is recognized in free text.
- Quality profile: share of empty values, duplicates, outliers, distributions and discrepancies between sources — as a repeatable report, not a one-time slice.
- Validation regulation: set of rules, trigger thresholds, run frequency and incident review procedure with responsible roles.
- Parsing and normalization configurations: rules for dates, addresses, phones and text fields, moved into separate versioned files.
- Maintenance instructions: how to connect a new source, where to change thresholds, how to roll back a rule if it produces false positives.
Plus a metric catalog: which metrics are built on top of the profile, from which fields they are assembled and how their comparability across periods is checked.
We hand over the kit with a walkthrough on live data: the customer's team runs the check themselves, changes the threshold and sees the result. Describe the task — we will come back with an estimate and a plan.
Case: quality assessment of an order warehouse
The customer is a manufacturer with direct sales: orders accumulated in a production PostgreSQL database for several years, and part of the history came in exports from previous systems.
Each analyst got a different revenue report — one filtered by order status, another by shipment date, a third by payment date.
Profiling the warehouse removes these disagreements in hours: it is clear which columns can be compared at all, and where comparison gives a false picture.
What profiling found
- In 17% of orders, the sum of line items did not match the total — from cents to several percent.
- 6% of order numbers were repeated: some were real duplicates after history migration, some were different orders from different legal entities.
- Dates were stored in four formats, time zones were not normalized, so orders "moved" between days.
- 23% of records had no customer identifier — they cannot be linked to a customer card.
- In 4% of rows, negative quantities occurred — returns recorded as sales.
What changed
The shares of discrepancies and duplicates were reduced to fractions of a percent; disputed rows are not deleted but flagged: history remains complete, and the revenue report is assembled from a single source.
Validation rules — line item reconciliation, order number uniqueness within a legal entity, unified date format, mandatory customer reference — were embedded into data loading.
A quality summary arrives after each update, so an error is reviewed on the day it arrives, not at the end of the quarter.
What to do if data is already deployed in storage?
There is no need to deploy new storage: profiling of an existing storage reads existing tables in read-only mode.
Access is granted to a separate account without modification rights; heavy passes run on a replica or with sampling — AI profiling does not require exporting data outside.
An audit of the current storage begins with a structural inventory: which tables are actually used, where entities are duplicated, whether keys match, whether there are violations of referential integrity.
Then for each column we calculate the share of empty values, number of unique values, distribution and discrepancy with the business reference data.
We restore semantics from names, comments, saved queries and ready-made reports: they show how analysts interpret entities in practice. This gives a classification — personal data, reference data, operational events — and turns disparate tables into a domain map.
The output is a prioritized issue register: field fill rate, format compliance, update lag, duplicates, competing sources of truth. With it, quality is measured regularly and compared across periods, and fixes are planned by impact on reporting.
A separate case is an inherited storage without descriptions: we assemble the picture from query logs, load scripts and report logic. To start with existing data, read access and a list of priority marts are enough — we will come back with a validation plan.
Technical concerns: questions and answers
Objections to data profiling usually come down to three things: whether automatic labeling can be trusted, whether data leaves the perimeter, and whether the result is reproducible on a repeated run. Below are answers for each point, with a view to how this is verified at acceptance.
Automatic labeling makes mistakes — how do you control this?
The profile is built on column statistics: distributions, cardinality, share of empty values, frequency patterns. Semantic labels are applied on top as hypotheses with confidence estimates.
Labels below the agreed threshold do not go into the report as fact, but enter a confirmation queue. We calculate accuracy on a control sample labeled by your analysts, and recalculate it on every rule change.
Does data stay within our perimeter?
Yes. Processing is deployed on your infrastructure; no access to external services is required either for label calculation or for profile storage. We provide only aggregated indicators and a glossary of concepts externally, without rows of source data. If you have segmented access between departments, the profile is built for each segment separately.
Will a repeated run give the same result?
The profile is a versioned artifact. We fix the version of rules and dictionaries, sampling parameters and source traversal order.
On an unchanged slice, the discrepancy between runs is reduced to zero, and when a schema or column type changes, the difference is visible in the report for a specific field.
Will profiling not change the data itself?
We work only in read-only mode: rights are granted for select, we do not request write access. If the source is under load, we limit parallelism and sample size, and move heavy passes to off-peak windows.
We fix the wording of thresholds, the list of labels and the confirmation procedure in writing before the start — this is the subject of acceptance, not an abstract "labeling quality".Let's discuss the task: how to start profiling
For the scope estimate to rely on facts, not assumptions, a small set of input data is enough at the start.
AI data profiling does not require exporting the entire database: a representative slice for a typical period is enough, showing both the structure and the nature of contamination.
Then we analyze the schema, calculate field statistics and compare values with reference data — and come back with a plan.
What to prepare for the task discussion:
- data slice — 2–3 exports as is, together with missing values, duplicates and "dirty" fields;
- list of sources: where records come from, how often they are updated, who is responsible for the system;
- business rules and reference data by which a value is considered correct;
- known problem areas — "dump" fields, outdated records, discrepancies between systems;
- privacy requirements: which attributes cannot be shown, whether anonymization is needed before analysis;
- expected result — quality report, semantics for the catalog or labels for models;
- responsible person on the customer side who confirms the rules and accepts the result.
This is already enough to estimate the scope of work and give a realistic project start date. Describe the task — we will come back with an estimate and a plan: what we do as the first step, which quality metrics we fix at the input and in what form we deliver the result.
