AI data catalog with autoclassification and PII detection

AI Data Catalog: Where Search and Access to Records Break Down

Fragmented repositories appear on their own: every team starts its own database, analysts start their own data marts, marketing keeps exports in spreadsheets on disk.

In a couple of years, the company has hundreds of tables, but field descriptions live in chat threads or in the head of an engineer who has already moved to another department. Formally, the data exists — finding it is impossible.

The cost shows up in daily work: searching across sources for the right table takes days because the name is abbreviated and the decoding is not documented anywhere.

An analyst asks in a general chat, waits for an answer, then builds a report from their own export — and gets numbers that diverge from the neighboring department. Duplication grows, trust in reporting falls.

Access to records is granted manually: request, approver, email, copy of the table by email. At the moment of granting, no one checks whether fields contain personal data and on what basis it is provided.

Revoking such access is also forgotten — it remains with the employee after moving to another project.

Ordinary search by names does not cure this picture. A catalog without table and field descriptions returns empty matches: the same field is called differently in five systems, and synonyms and relationships between tables are not defined anywhere.

We break down how this is closed: how an AI data catalog restores descriptions automatically, links tables to each other, and turns permission issuance from chat threads into a verifiable process.

Autoclassification and Labeling: Measurable Effects of Implementation

Autoclassification starts not with a model but with inventory: the catalog scans repositories and databases, collects schemas, column statistics, and sample values. Then each field gets a label — entity type, sensitivity level, owner, source.

Labeling runs in two modes: by rules (dictionaries, column profiles, value patterns) and by the meaning of the data itself, where rules produce false positives.

The first measurable effect is source discovery time. When a field is labeled, an analyst finds the right table by querying the label rather than chatting with three teams: the typical path shrinks from hours to minutes.

The share of labeled records works as a completeness metric — on critical domains it reaches 90–95%, and the remainder goes to a manual review queue instead of sitting as "unknown."

The second effect is data access control. The sensitivity label becomes a policy attribute: queries to personal data fields go through masking or separate approval, and export of an unlabeled dataset raises a warning.

Rights stop being granted "just in case" because it is visible what exactly lies in the table.

What stops breaking is the class of incidents "leak through unlabeled data": a database copy for tests, an export to a report, a log with extra fields.

Categories for labels can conveniently be taken from external requirements: GDPR sets the framework for personal data, and labeling ties to it rather than to internal agreements that diverge between teams.

We verify the effect in a short pilot: two or three repositories, a report on the share of labeled fields and the number of closed access items before and after.

Modes of Autoclassification and Personal Data Discovery in the Catalog

A manual registry holds while there are a dozen sources and schemas rarely change. After that, it diverges from reality: a column was renamed, a table with customer addresses appeared, but the registry remains as before.

Personal data discovery using such a document gives false calm — findings are on paper, but not in the repository.

The choice comes down to four working options: manual registry, ready-made platform, in-house development, and hybrid. Autoclassification modes differ not by algorithm but by who maintains the labeling rules and how quickly they catch up with schema changes.

Hybrid is chosen more often than others in practice because it separates responsibilities: the platform maintains accounting and relationships between objects, while custom detectors cover industry patterns — tax IDs, policy numbers, internal identifiers.

Resilience comes not from the catalog itself but from regular source scanning, rule versioning, and labels that carry over to new tables when the schema changes.

Implementation Stages: From Source Audit to Handover of Policies

Catalog implementation starts not with installation but with inventory: until it is known what lies where, any labeling will be a guess.

The order of work is structured so that each stage relies on the previous one, and the first measurable result appears before rollout to the entire loop.

1. Source audit. We collect a list of databases, repositories, queues, and external exports, record owners, update frequency, and access method. The output is a priority map: what to connect first, what to defer and why.
2. Relationship schema between tables. We analyze keys, reference tables, and relationships between entities, mark duplicates and discrepancies in the interpretation of the same fields. Without this, the catalog will show disconnected sets instead of a connected picture.
3. Labeling rules setup. We describe domains, tags, and sensitivity levels, including personal data detection rules. Rules are versioned — changes in logic can be rolled back and explained.
4. Validation on historical data. We run the rules on accumulated datasets and compare with manual labeling of a control sample. We analyze false positives: adjustments usually concern thresholds and exceptions for individual fields.
5. Pilot on one domain. We connect a limited loop and look at real load, metadata search speed, and quality of query responses. Mistakes here are cheaper than after full implementation.
6. Catalog implementation and policy handover. We roll out to the remaining sources, set up regular metadata updates, and hand over policies to the customer team: how to add sources and change labeling rules.

Documentation, Access Policies, and Schemas That Remain with You

A data catalog that depends on one engineer is a quiet risk: after they leave, labels stop updating, disputed rules are forgotten, and trust in the catalog falls. Therefore, along with a working loop, we hand over everything needed to develop it on your own.

• Labeling configurations. Features by which a field is assigned to a category, configured confidence thresholds, and the procedure for adding a new category without rebuilding the catalog.
• Field descriptions and glossary. A unified dictionary of terms — so the same entity in different systems does not diverge in values.
• Relationship schema. A map of data movement from source through transformations to report: immediately visible which reports a change will affect.
• Role matrix. Domain owner, description editors, access approvers, and the entry point for a disputed label.
• Access policies. Permission granting and masking of sensitive fields by sensitivity labels, with inheritance across nesting levels.
• Catalog update policy. Rescan frequency, connecting new sources, and removing outdated records with responsible parties specified.
• Team instructions. How to label a new source, dispute an incorrect label, and provide data to an analyst without removing restrictions.

Artifacts remain in your infrastructure alongside code: changes go through normal approval rather than living in chat threads.

Case Study: How Order Was Brought to Fragmented Sources at a Retail Chain

A retail chain with an online store, loyalty program, and warehouse loop kept data in six repositories. Table passports were maintained manually in a registry table, and by the start of the implementation case, 1,800 of 2,400 tables remained without description and owner.

An analyst searched for the needed field for up to three days, and some reports were built from personal copies of exports.

We connected Unified Catalog to each source: connectors capture schema and column statistics, profiling calculates the share of empty values and uniqueness.

Automatic labeling runs in two passes — rules by names and relationships (identifiers, reference tables) and a content classifier that assigns the table to a domain: sales, loyalty, logistics.

Personal data detection by patterns and context marks sensitive columns and assigns an owner by source domain.

Metrics Before and After

Disputed columns are not labeled silently — they go to a confirmation queue for the owner, so the catalog does not diverge from reality.

A new source goes through the same connectors and rules: no separate integration is written for it, and access incidents are closed before permissions are granted.

What to Do If Source Accounting Is Already Manual

We do not throw away the manual registry — it becomes the basis for initial catalog population. We export tables into machine-readable form and analyze columns: owner, purpose, field composition, retention period.

Each row is matched against actual schemas in repositories — immediately visible where accounting diverges from reality.

Description migration happens in two steps: text comments are transferred as draft annotations, while autoclassification independently scans the data and suggests labels, including personal data indicators.

Discrepancies between manual and machine marking are not averaged — they go to owner confirmation, who knows what actually lies in the field.

The transition period is built on dual accounting: the registry lives as before, the catalog is populated automatically, and discrepancies are collected in a separate report. Departments work as usual — nothing is blocked or rewritten retroactively.

Record reconciliation runs on a schedule and answers two questions: which sources exist in the systems but are absent from accounting, and which are listed in accounting though they no longer exist.

For conflicts, a rule is defined in advance — by freshness of change or by owner level — otherwise every pair of discrepancies turns into a dispute.

The exit criterion from the transition period is measurable: the share of uncovered sources and the share of records without a confirmed description.

When both remain at zero for several reconciliation cycles, manual input switches to suggestion mode, and dual accounting is removed without losing agreements.

Implementation Concerns: Access, Labeling Quality, Support

The catalog connects to sources under a service account with read-only access to metadata — already configured rights and roles in the systems themselves are not rewritten.

Visibility is inherited from your user catalog: if an employee could not see a table before, they will not see it in search either. Fields with personal data are closed by a masking policy, so descriptions do not reveal contents.

We check labeling accuracy on a reference sample: model answers are compared with manual labeling and the false positive rate is calculated separately for each data class.

The confidence threshold is configured, disputed cases go to the domain owner for confirmation rather than into the description automatically. Labeling is exported in open formats — metadata transfers to OpenMetadata and similar catalogs without rebuilding.

How many false positives will there be?

At the start, we configure thresholds on your data; rare classes are first marked as disputed. Then we add rules and retrain on confirmed examples; dynamics are recorded by a before-and-after report, not just as a claim.

Who updates descriptions after implementation?

When the schema changes, fields are marked as outdated, and descriptions are rebuilt on a schedule. The update policy, domain owners, and escalation procedure are handed over via documentation; post-implementation support is a separate mode with an agreed response time.

Let’s Discuss the Task: We’ll Select a Mode and Return with a Plan

For the estimate to be substantive rather than "about six months," we need access to the real data picture, not a description in general terms.

Tell us where the sources are located, who owns the systems, and what has already been done on labeling — that is enough to return with an implementation plan and clear stages, not a range "from and to."

What you send as input — this is enough to estimate the work:

• list of source systems: databases, file storage, queues, external integrations — with schemas and volume order;
• 2-3 anonymized slices of real data for each major source: we use them to check autoclassification quality and rule resilience;
• examples of what is considered personal data in your loop: documents, contacts, payment details, indirect identifiers such as a login and address combination;
• current access rules and roles: who can see catalog metadata and who can see the data itself;
• current glossary of terms and tags, if the catalog is already partially maintained — we do not reinvent the dictionary;
• journal requirements: what is recorded for each data access and for how long it is stored.

During the review, we will discuss the mode of working with sources, metadata storage locations, and the points where classification and personal data discovery integrate into existing processes.

Based on the results, we will return with a stage-by-stage estimate, an implementation plan with checkpoints, and team composition. Describe the task — we will arrange a substantive conversation.