Why graph neural networks lose accuracy on live data
On a holdout sample, graph neural networks show high quality, but in the production loop the model's accuracy on real data drops — sometimes by half. The reason is almost never the architecture: the "data — graph — model" link breaks.
First is edge labeling. Edges are labeled fragmentarily: confirmed connections exist for a small share of nodes, while the rest is inferred by heuristics based on matching attributes.
The model trains on a biased subsample and confidently errs where the connection is weak — and it is precisely such areas that determine graph behavior in production.
Second is structural drift. The composition of nodes and edge types changes: new roles appear, and some old connections stop generating events.
The topology is rebuilt gradually, metrics slip unnoticed, and the discrepancy is discovered from decision quality rather than from monitoring.
Third is the mismatch between the export and the event stream. Training uses a batch snapshot with aggregates and a daily delay, while in operation events arrive late and duplicated.
Features are calculated over different windows, and the graph at the model input looks different from the one used in training.
Therefore, data graph preparation is not a one-time stage before training but part of the cycle: reconciliation of the connection schema, monitoring the share of labeled edges, and unified rules for recalculating features for training and the production loop. We explain how this is addressed.
What implementation delivers: impact in measurable terms
A graph model relies not on an individual record but on connections between entities, so the impact of implementation is visible in metrics, not in feelings.
We check the five points below before the start and record them after deployment — on a holdout sample and on live traffic.
- Training and retraining time. Recalculation runs over the affected subgraph, not the entire database: an update after new connections appear takes minutes, and full retraining takes hours instead of days, with no downtime window.
- Quality on rare classes. Neighbor features supplement scarce examples, so the false positive rate on rare events falls, and recall grows where tabular models lack enough statistics.
- Robustness to new nodes. A new entity receives a score immediately based on its connections, without accumulating history or manual labeling.
- Eliminated failure class. A class of errors that rules did not catch goes away: anomalies through chains of indirect connections rather than through a single feature.
- Maintenance load. Some manual rules are replaced by graph structure — fewer engineering hours are needed to support and tune thresholds.
We measure the impact on infrastructure where graph layers are built on PyTorch Geometric: a unified feature pipeline for training and inference, so metrics are comparable across iterations.
Training and inference modes: what fits the task
The training mode is chosen not by the "novelty" of the method but by how often the graph itself changes and how much time is available for a response.
Training on historical exports gives a reproducible result: one data snapshot, identical metrics, a clear rollback point. Fine-tuning on new connections is needed where the structure changes daily — otherwise the model becomes outdated.
Batch inference calculates scores for the entire snapshot at once — convenient for regular reporting. Streaming inference responds to an event at the moment it appears but relies only on the connections that have already reached the system. A hybrid with strict rules covers areas where a model error is unacceptable.
| Option | When it fits | What limits it |
|---|---|---|
| Training on historical exports | Connection structure is stable, reproducibility matters | Data becomes stale between rebuilds |
| Fine-tuning on new connections | The graph grows daily, labeled fresh examples are available | Without drift control, errors carry over into subsequent versions |
| Batch inference | Scores are needed regularly, on a schedule | No reaction to an event at the moment it appears |
| Streaming inference | A decision is needed immediately, at the moment of the event | Sees only known connections, weak start on new nodes |
| Hybrid with strict rules | There are areas where an error is unacceptable | Rules require maintenance and revision |
In practice, modes are combined: training uses an export, inference is streaming, and rules safeguard critical decisions.
Work stages: from data audit to model handover
A graph model begins not with code but with an inventory of entities and the connections between them. Below are the seven stages the project goes through: from source audit to handover of a trained model with a reproducible retraining pipeline.
- Data audit. We examine the sources: in which tables, logs, and services entities reside, and in which ones the connections between them reside. We record gaps, duplicates, outdated records, and update frequency.
- Graph schema design. We define node and edge types, connection direction, weights, and timestamps. We reconcile the schema with domain logic: a connection that does not exist in the real process will not appear in the graph.
- Node and edge feature preparation. We bring attributes to a unified form, encode categories, normalize numeric fields, and add aggregates over neighboring nodes.
- Training setup. We select an architecture for the task — node classification, link prediction, or ranking — as well as propagation depth and regularization.
- Holdout validation. We measure quality on a period that did not participate in training, with a time-based split: a random split causes leakage and inflated numbers.
- Comparison with the current solution. We run the model and the existing logic on the same dataset and show the difference in metrics, not just their absolute values.
- Acceptance and handover. We deliver weights, training configurations, the graph schema, graph assembly scripts, and documentation; we record the achieved metrics and the retraining policy.
What remains with you: schemas, configurations, policies
We hand over not a final report but a working kit that your team can use to train, validate, and extend the graph model without us.
We review each artifact on a call with engineers and run one retraining cycle on your side — so the policies do not remain a document that no one opens.
- Graph schema description: node and edge types, attributes, where each field comes from, update rules and frequency — with examples of exports on real data.
- Training configurations: hyperparameters, the training/holdout split, random seed fixing — the run is reproducible and yields comparable metrics.
- Model documentation: what exactly it predicts, on what data period it was trained, where the applicability boundaries are, and which error classes are already known.
- Retraining policy: triggers (share of changed connections, feature distribution shift, metric drop below threshold), frequency, who launches it, and how the result is accepted.
- Degradation response policy: control metrics, rollback procedure to the previous weight version, decision owner, and response time.
- Access control: roles for reading raw graph data and for predictions only, request logging, and the procedure for granting rights to new employees.
- New version release instructions: validation on a fixed set of examples, acceptance criteria, and the procedure for replacing the model in the production loop.
Everything resides in your repository and your storage — access remains with you; we do not keep copies ourselves.
Case study: how we cut the false positive rate in half
The implementation case study is identifying hidden connections between counterparties: shared details, phone numbers, addresses, and payment chains through shell links.
The original system checked entities using tabular rules and manual thresholds, each record separately, and pulled connections via exports. The signal triggered on a formal field match, and the false positive rate reached 58% — analysts manually reviewed six out of ten cases.
What we changed
The data model was moved to a graph: entities became nodes, while transactions, contracts, and shared details became edges with a type and timestamp.
Identifiers were normalized and duplicates collapsed so that one counterparty would not split into several nodes; graph storage was built on Neo4j.
We calculate features over a neighborhood of 2-3 hops rather than a single record, so intermediaries and second-level matches are included in the risk assessment.
Metrics before and after
| Metric | Before | After |
|---|---|---|
| False positive rate | 58% | 27% |
| Recall on analyst labels | 0.71 | 0.89 |
| Depth of discovered chains | 2 nodes | 5 nodes |
| Review of one signal | hours | minutes |
A separate effect came from training on the labels analysts accumulated over time: the model finds some connections on its own, without a predefined rule.
Graph models or tabular models: what to choose for the task?
The choice between tabular and graph models rarely comes down to algorithm quality. It comes down to the question: does the answer for an object depend on whom that object is connected to, or is it enough to describe the object by its own features? If the connection carries signal, the graph provides uplift; if not, it only adds maintenance cost.
A tabular model sees a row: object features with known values. A graph model adds a second channel — edges, over which neighbor features are aggregated at each layer, and the object receives context from its environment.
In anti-fraud, recommendations, supply chain analysis, and finding related counterparties, this gives a noticeable gain: patterns invisible in individual records are visible as structure.
The flip side is data. The graph must be rebuilt regularly: the node and edge schema, the order in which connections appear, their quality, and missing rates.
An error in edges propagates to all layers and degrades the result more quietly than an error in features, and explaining the answer is harder — it is not always clear which neighbor influenced the score.
The practical selection order is as follows. First, we build a tabular model and add connection-based aggregates to it: number of neighbors, category shares, neighborhood averages. Such a set often covers most of the possible uplift at lower maintenance cost.
We take the graph approach if the residual error is systematically related to structure — failures are concentrated in dense groups of connected objects.
We test the hypothesis on a single holdout snapshot, comparing both approaches by the target metric. We decide in favor of the graph only when the uplift is stable across different periods and on new nodes, not only on those that were in training.
Technical concerns: robustness, data, support
Below are four questions most often heard at acceptance: how the model behaves when new nodes appear, what to do with a small amount of labels, how to survive a source schema change, and what remains with the team after handover. We answer about mechanisms, not promises.
What happens when new nodes and connections appear in the graph?
The model operates in inductive mode: a node representation is assembled from its features and those of its neighbors, so new vertices do not require retraining.
Their representations are computed quickly, and a full recalculation is needed only when the connection distribution shifts — this is caught by monitoring the share of new edges and connectivity.
Is there enough labeling if there is little of it?
The connection structure itself carries signal, so training uses partial supervision: we label the most informative nodes rather than labeling in a row.
Before starting, we assess how much labeling is sufficient for the target quality — otherwise labeling consumes time without return.
The source changed its data schema — will the model have to be rewritten?
No. The layer that transforms the source into a graph is separated from the model: the field mapping changes, while the core remains. The input contract is checked at the schema level, so a type mismatch stops loading before rollout, not after it in production.
What remains with us after project handover?
Configuration, feature schema, retraining policy, and a drift monitoring dashboard — this is enough to maintain the model on your own.
We work on an open stack with a live community, including PyTorch Geometric: engineers for such a loop can realistically be found on the market, without being tied to us.
Let's discuss the task: we will return with an estimate and work plan
To ensure the project estimate is based on facts rather than assumptions, a task description and access to a data snapshot are enough to start.
What is key for graph neural networks is not the volume of tables but the structure: which entities become nodes, which connections between them are significant, whether labels exist, and how often the graph changes. Please send:
- task description in free form — what prediction or ranking is needed as output, who uses the result and how;
- data schema — tables or exports with what you consider nodes and edges, including reference data and many-to-many relationships;
- a real data snapshot over a representative period, with gaps and noise, not a cleaned sample;
- labels, if available: how they were formed, how many classes there are, how balanced they are;
- the current benchmark — how the task is solved now (rules, heuristics, manual work) and which metric you consider acceptable;
- environment constraints — where the solution must operate: an isolated environment, your own servers, or an external environment, and what requirements exist for access and logging.
In response, we will return with a project estimate: a realistic timeline range, team composition, and a list of risks indicating how we address them.
Next — a staged work plan with checkpoints, a proposal for a pilot launch on your data snapshot, and the terms of solution handover: documentation, reproducible environments, and training for your engineers. Describe the task in any convenient form — we will clarify the rest ourselves.
