AI Data Lineage Tracking: Automation and Implementation

Data Lineage: Where the Data Provenance Chain Breaks

A quarterly report shows revenue several percent lower than in the accounting system. The analyst climbs the chain: the mart layer, intermediate tables, loading jobs, external reference data — and already at the third or fourth step the trail breaks.

It is impossible to determine which transformation changed the value, at what point, and why: data lineage is not recorded anywhere.

The problem is not one error. The value is distorted by deduplication during table joins, type casting, manual edits in reference data, and reloading an increment for a past period. Each such step changes the data but leaves no "input — transformation — output" link. Data provenance is reconstructed manually, from correspondence and code comments.

The cost is not a single incorrect report, but the loss of trust in the number. Source auditing turns into a multi-day investigation, and during an external review there is simply no way to confirm the data provenance for a specific report line.

This is solved by automated data lineage: links between fields are built from the queries and metadata themselves, not from documentation, and a break in the data chain is immediately visible. We explain how it works and what it gives the customer.

What Automated Data Provenance Tracking Provides

Automated data tracking is built on intercepting events where data actually moves: in the orchestrator, in the job queue, in the transformation layer.

For each run, an "input — operation — output" triple is collected, and from these events a graph of relationships between tables, reports, and marts grows by itself. Maintaining such a map manually is pointless: it becomes outdated after the very first schema change.

The first measurable effect is incident investigation time.

When a discrepancy surfaces in a report, the question "where did this come from" is resolved by traversing the graph backward: you can see which sources feed the mart, which run last wrote to it, and what changed between builds.

Investigation shrinks from hours of cross-team correspondence to a few minutes and a single query to the graph.

The second effect is classes of failures that stop reaching the customer.

Renaming a column, changing a field type, swapping a mart's source, duplicating a load after a restart — all of this is caught during validation, not at the stage when "the numbers in the report don't add up." At the same time, the share of reports with a confirmed source grows: coverage becomes a number, not a feeling.

Technically, it is convenient to build the data observability layer on open formats such as OpenLineage: data lineage events are described by a single schema, so the graph is not rewritten when the stack changes.

What remains is to agree on discipline — every report has an owner, and every mart has a declared source.

Data Lineage Implementation Formats: Comparing Approaches

The approach to implementing data lineage is determined not by the tool, but by where the transformation logic lives.

If the rules are described in a repository, code parsing works; if some reports are assembled manually in an interface, without in-stream tracking the graph will remain incomplete.

In practice, the modes are combined, and they are chosen based on data maturity: the more teams and releases there are, the less trust there is in a manual source inventory.

The maturity guideline is simple: manual inventory covers the starting level, code parsing — when transformations are centralized, in-stream tracking and hybrid — when there are dozens of sources and the cost of an error in a report is high.

How We Build Tracking: Stages of Engineering Work

We start with a source audit: we go through storages, queues, services, and job schedulers, and record where a dataset is born and where it is transformed.

The output is a list of owning systems, access methods, and areas where transformation logic is described only in code or in engineers' heads.

1. Source audit. Inventory: which systems write, which read, who owns them, what access is needed. We mark the points from which metadata can be collected automatically.
2. Data map. We agree on the graph model: nodes are datasets, reports, marts; edges are operations and dependencies. We agree on the level of detail — down to the column or the table.
3. Collection setup. We connect log parsing, query interception, and orchestrator metadata. Where automation cannot see the source, manual description with a mandatory owner.
4. Graph validation. We compare the model with reality: a backward pass from the final report to the source record and checksum reconciliation on a sample. Discrepancies go into a separate list.
5. Pilot domain. We take one flow and run the entire loop on it: we look at the share of nodes covered by links and how long incident investigation takes.
6. Regulations and handover. We describe how the map is updated and who is responsible for new sources, and hand over access, documentation, and team training.

Handover Artifacts: Schemas, Regulations, Configurations

We hand over not an archive of documentation, but a working environment that your engineers can develop without us. Everything resides in your repository and your infrastructure, and access is set up for your accounts, not ours.

• Collection configurations: sources, schedules, incremental loading strategies, handling of delays and retries.
• Graph schema: nodes and edges, types of relationships between tables, processing jobs, reports, and marts, rules for automatic enrichment.
• Data dictionary: mapping of business terms to physical objects, domain owners.
• Access policies: roles, visibility segmentation by domain, procedure for granting rights to new objects.
• Update regulations: how a new mart or job enters the graph, who confirms the relationship, what to do when a field is renamed or a source changes.
• Integrity monitoring: checks for broken links, notifications to responsible parties, a list of common causes of false positives.
• Documentation handover: model description, examples of tracing metric provenance, a checklist for diagnosing discrepancies.

During acceptance, we go through each item using live queries: your specialists themselves build the path from the report to the source, add a new object to the graph, and edit the access policy.

Open questions and areas of responsibility for each part of the environment are documented in writing.

Case Study: Incident Investigation Reduced from Days to Hours

Starting Point

Investigating a single discrepancy in reporting took two to three days.

Table owners were interviewed manually, and the data provenance graph (data lineage) was reconstructed from correspondence and orchestrator run logs — with a dozen reports per week, this consumed almost half of the data team's time, and the root cause was not always found, and the incident returned a month later along the same path.

What We Changed and What We Got

We tagged provenance at the level of processing jobs: collection, transformation, and loading events are written to a single catalog, where every table and field is linked to a source, owner, and specific run.

The open tagging standard OpenLineage became the foundation — existing jobs did not have to be rewritten. On top of the catalog, we set up alerts: a schema or field type change at the source raises a signal before the discrepancy reaches the mart.

The average incident investigation time dropped from two days to three hours; typical cases — a lost field address, a broken type — are closed in 40 minutes.

The share of sources under observation grew from a third to 94%, and the share of investigations without manual owner outreach — from one in five to four in five.

The cause is now recorded in the incident card, and a repeat case along the same path is investigated using a ready-made template.

"Previously, when asked where this number in the report came from, I would answer by the evening of the next day. Now I open the provenance graph and answer in five minutes — most often before the question reaches management," — a data engineer on the customer side.

How Code Parsing Differs from In-Stream Data Tracking

Code parsing and in-stream tracking answer different questions, and substituting one for the other is expensive. The first reads source code, configurations, and templates and builds a dependency map based on how data accesses are written. The second looks at actual operations during execution.

Code parsing provides a picture before execution: it sees even branches that are not executed today, finds reads from sources and writes to marts, and builds the graph in full.

The weak spot is dynamics: if a source name is assembled from parameters or substituted from a reference, static parsing leaves a placeholder node, and the link is lost.

In-stream tracking is accurate by fact: it records every completed read and write along with the job, executor, and time.

It pays for this with coverage — rare scenarios and unexecuted branches are not included in observation, and for jobs that have not been restarted for a long time, the picture becomes outdated.

Usually, a mixed approach closes the task: static analysis provides the map skeleton and near-complete coverage, while the stream confirms links and serves as the source of truth for dynamics.

Discrepancies between the two sources are highlighted as anomalies — this is how both "dead" code and undocumented data exchange are found. The share of links with a confirmed source grows from about two-thirds to nine out of ten.

Technical Doubts Before Starting Work

Before starting, the customer's engineer checks not the presentation but the system's behavior at the edges: what happens with an undocumented source, who fixes the provenance graph after a schema change, whether it is realistic to start with one domain instead of the whole platform. Below are five questions asked at almost every review, and answers without marketing.

What happens to the graph when the schema changes — renaming, a new column, a drop?

Changes are recorded as events, and the graph is stored by time slices: you can see what the chain looked like before and after the migration. Renaming does not break links — the previous edge remains in history and is marked as requiring owner confirmation.

Can we start with one domain instead of the whole platform?

Yes, and that is usually what we do. The first domain provides a working graph and validated parsing rules; after that, onboarding goes faster because common nodes are already described. You get a result on a narrow scope before all teams are involved in the project.

Will connecting sources require changes in production?

No. We work in metadata and log read-only mode, write nothing to your storages, and do not change pipelines. If a source is closed, we agree on access to the catalog or a statistics snapshot — that is enough to build links.

What remains with us after handover?

The graph schema, parsing rules, the procedure for responding to schema changes, and instructions for the on-call team. We separately discuss which nodes require an owner and conduct knowledge transfer — after it, further development does not depend on us.

Let's Discuss the Task and Come Back with a Plan and Estimate

To come back with a work plan and a realistic timeline estimate, rather than a range "from and to," we need a short brief on your data environment.

Describe the task in free form: which reports and models should be covered by data lineage first, what is already collected from metadata, and at which junction the picture is lost — between sources, in transformations, or at the marts.

On the customer side, the following is usually enough for estimation:

• list of sources, intermediate layers, and target marts of the first circuit;
• scheduled jobs and processing schedule — where transformations are formed;
• list of accesses: read access to the metadata catalog, storage systems, and repositories with SQL;
• data owner who will confirm the correctness of the tagging during acceptance;
• requirements for graph detail and the retention depth of change history;
• environment constraints: closed segments, personal data, regulatory requirements;
• existing flow schemas and table descriptions, if they are maintained.

After the review, we will come back with the composition of stages, a timeline estimate, and a list of accesses that need to be opened for each stage. Describe the task — we will discuss the details and agree on the start.