Why neural network solutions do not reach the production environment
A demonstration of a neural network solution usually takes place on a prepared dataset: clean examples, even conditions, neat labeling. Metrics on such a sample look convincing, and the solution goes into the production environment.
There the picture changes: quality drops on the live stream, and the customer again sits down to check the result manually — exactly as before the project.
The reason is that the demonstration diverges from real data. In the live stream there is a different class balance, noise, truncated and duplicate records, and some scenarios did not occur at all during training.
On the holdout set and in the stream, the share of correct answers diverges noticeably, and for individual classes — by multiples. The model answers confidently even where it should have refused, so manual checking returns as a permanent stage, not as a temporary measure.
The second gap is organizational. After launch, the model has no owner: no one to track data drift, replenish the reference labeling, and retrain on fresh examples.
Quality slowly slides, trust in the system falls, and the pilot launch does not reach implementation — the solution is quietly turned off.
This is closed not by replacing the model, but by an engineering loop around it: a reference set with live examples, quality thresholds, control of the refusal share, an update procedure, and a person responsible for the model. We break down how this is arranged.
Measurable effects: processing time, error rate, downtime
Five areas by which we calculate the effect: processing speed, share of automatic decisions, class of eliminated errors, resilience to overload, offloading specialists.
We take measurements before and after on the same task stream, and build dashboards in Grafana — both the customer and the on-duty shift see the dynamics.
Processing time for a request
We look not at the average, but at the 95th percentile: it shows what the user feels at the tail of the queue. Caching repeated answers and parsing heavy requests in the background reduce processing time by multiples on typical requests.
Share of decisions without manual review
We divide the stream into three corridors: auto-closing at high model confidence, specialist review at medium, manual handling at low. We select the threshold using the error matrix — the share of decisions without manual review grows, quality does not fall.
Eliminated error class
What goes away is not "an error in general," but a specific class: lost fields during transfer between systems, duplicate requests, typos in details. We close it with a regression set so that it does not return with the next model version.
Resilience to failures and overload
The queue smooths peaks, retries with increasing delay extinguish brief failures of external services, and during degradation the stream moves to a simplified scenario. Shift downtime due to model unavailability disappears from the picture.
Offloading specialists on repetitive tasks
Verification of details, sorting requests, and initial responses go to the model. The freed-up time goes to complex cases: we measure by the number of processed requests per shift and by the share of returns for rework.
Implementation approaches: ready-made model, fine-tuning, custom development
The implementation mode is chosen along two axes: how much labeled data there is and how much the task differs from a typical one. A ready-made model without fine-tuning gives a result immediately, but the quality ceiling is set by someone else's dataset.
Fine-tuning raises the metric on your vocabulary; custom development is for when ready-made architectures do not reproduce the response format.
| Option | When it fits | Limitations |
|---|---|---|
| Ready-made model without fine-tuning | Typical tasks: request classification, entity extraction, speech recognition on general vocabulary | Does not know the domain — quality remains baseline, behavior cannot be configured |
| Fine-tuning on your own data | Several thousand labeled examples and a stable input format | Labeling is needed; when data changes, training is repeated; risk of overfitting on a narrow sample |
| Model development for the task | Your own metric: special response format, several data types at the input, speed requirements | Longer to put into operation, requires your own training loop and quality control |
The second axis is where the model runs: calling an external service or deploying in an internal environment. An external service starts faster and does not require your own capacity, but data leaves the perimeter.
An internal environment meets data processing requirements, but training and putting into operation fall on your infrastructure.
A neural network can be excessive: if a rule is described by a reference book, deterministic logic gives the same accuracy and an explainable result.
This is checked at the assessment stage — we compare options by the quality metric and say directly if a simpler mechanism solves the task.
Work stages: data audit, design, setup, validation
The order of stages is not arbitrary: the quality ceiling of a solution is set by data and processes, not by the model architecture.
If you skip the audit or the integration design, you get a model with good demo metrics that in the customer's production environment either does not receive the necessary input data or disrupts the existing process. Therefore, each step ends with an artifact that can be verified before moving to the next.
- Data and process audit. We look at which events are already logged, with what delay and completeness, where there are duplicates and gaps, what share of operations is performed manually. The output is a map of sources and a list of what will have to be added or cleaned.
- Solution design and integration points. We determine where the solution fits into the process: a synchronous response to a request, background processing of a batch, or a hint to the operator. We fix exchange formats, behavior when the service is unavailable, and fallback to the previous logic.
- Data preparation and labeling. We agree on labeling rules and handling of disputed cases, and check the consistency of several labelers. We split into training, validation, and holdout sets by time, not randomly — otherwise neighboring records leak between sets and the evaluation is inflated.
- Baseline. First we assemble a simple solution and measure the current process; otherwise it is unclear what exactly the model complexity adds.
- Setup for the customer's environment. Fine-tuning on domain data, response time limits, caching of repeated requests, control of data drift when the stream changes.
- Validation on the holdout set. We look not at the average metric, but at the breakdown by segments and the worst cases; we fix the quality threshold and the rule "if uncertain — hand over to a human."
- Solution handover and team training. Code, configurations, deployment design, retraining procedure, and incident analysis — so that support does not require us to be constantly available.
The final step closes the main implementation risk: a solution that works only for its authors. The customer's team must be able to restart it, retrain it, and explain the result.
Project artifacts: designs, configurations, procedures
The project is handed over not by showing a working prototype, but by a set of artifacts that allow the system to be operated, improved, and transferred to another team.
Everything hardcoded in code and settings is recorded in documents — the solution does not remain tied to the implementers who assembled it. The set is transferred to your repository and knowledge base with administrator rights.
- Solution architecture design — data flows from source to model response, service boundaries, external dependencies, failure points, and degradation procedure if an adjacent system is unavailable.
- Configurations and environment parameters — which values are set for development, test, and production environments, what changes during migration, where secrets are stored and how they are rotated.
- Data access policies — which employees and services can see the source data, at which step depersonalization is performed, how rights are granted, limited, and revoked.
- Quality control procedure — a set of checks before rollout, thresholds for the share of correct answers on the control set, and conditions under which an update is not released.
- Operations instructions — typical operations with step-by-step descriptions: start and stop, retraining on new data, failure analysis, rollback to the previous version.
- Metrics log — which indicators are captured, at what frequency, where to view dynamics, and by what signs it is visible that quality has drifted.
Each artifact is accepted together with the working system: we show how to use it on a real task, not just hand over files. If your team makes improvements, these documents remain a reference point — they show what is configured this way and why.
Case study: request sorting before and after implementation
Requests from the form, email, and messengers fell into one common inbox, and operators manually sorted them into 14 areas. About a third of the stream was processed automatically — only where the topic was indicated in the email subject.
The first response went out on average after several hours, some requests hung until the end of the day, and the share of manual rework after labeling reached half: the topic had to be reassigned, duplicates had to be merged by hand.
What was changed
We built a classifier based on the open model Mistral, deployed in the customer's environment: the text is split into fragments, each receives a topic, priority, and duplicate flag, and the routing rule sends it to the required queue.
The model was fine-tuned on a year and a half of history — this gave resilience to wording that is not in the operators' instructions. Manual labeling was kept as a reference for regular quality checks.
What changed
About 92% of requests are processed automatically, the time to first response decreased from hours to minutes, and the share of manual rework fell to approximately 5%.
The remaining percentages are non-typical requests: complaints with emotional coloring, emails without text with attachments, requests on two topics at once.
The system does not try to guess such cases — it lowers confidence and routes the request to an operator along with a transcript and the customer's history.
How neural network solutions differ from ordinary automation
Hard rules are deterministic code: conditions, dictionaries, and thresholds where the answer follows unambiguously from the input.
It is explainable and covered by tests, but becomes more expensive with data variability — each new wording or format is a separate branch to maintain.
A trainable model instead of conditions receives examples and produces a probabilistic score: it works on input that was not described in the rules, but does not guarantee "always correct."
| Option | When it fits | Limitations |
|---|---|---|
| Hard rules | Features are formalized, input is homogeneous, decision explainability is needed | Breaks on variability, combinatorics of conditions grows |
| Trainable model | There are many features and they are non-formalized, input changes | Labeling is needed, quality is probabilistic, drift control is required |
| Hybrid | Critical parts — by rules, fuzzy input — by model | Support is more complex, an arbitration layer is needed between the environments |
The quality threshold is fixed before implementation, not after: on the holdout set we calculate precision and recall, separately assess the cost of errors of different classes, and choose the confidence level below which the answer goes to manual processing.
This is a standard mode, not an emergency one — the share of such requests is measured, not hidden. If the model covers 85–90% of the stream, the remainder goes to the operator with prepared hints.
Drift is the second reason a model cannot be turned on and forgotten: the input distribution changes, and metrics on the stream diverge from test metrics.
Therefore, we version models and datasets in a registry like MLflow, monitor the input, and keep rollback to the previous version — reverting takes minutes and does not require rebuilding the entire environment.
Rules are cheaper when there are few features, the input is predictable, and the decision must be explainable: arithmetic in documents, a fixed set of statuses, verification against a reference book.
A model pays off where input variability is high and describing it with conditions is more expensive than labeling a sample.
Technical concerns: environment, data, model support
Technical concerns usually surface not at the start, but at acceptance — when it turns out that the model has nowhere to be deployed or it is unclear how to update it. Below are four questions from technical sessions and our default answers.
Can the solution be deployed in an internal environment?
Yes. We assemble containers and deploy them in the customer's isolated network: without outbound requests to the outside, updates via an internal image registry. Data and model weights do not leave the perimeter, and we agree on hardware requirements before the start.
What if there is not enough data for training?
In that case, we do not start training from scratch. We take a pretrained model, fine-tune it on a small sample, and expand labeling with active learning and augmentation.
The guideline is hundreds of labeled examples per class; if there is no data at all, we start with rules and connect the model as an auxiliary signal.
How is quality checked before handover?
We fix acceptance criteria before the start, then calculate metrics on the holdout set, analyze the error matrix by class, and manually check disputed cases.
Separately, we look at resilience on data that was not in training — this way overfitting is visible before errors reach users.
What does model support after launch include?
We monitor data drift, the share of uncertain answers, and metric degradation. Artifacts are versioned, so rollback to the previous version is one operation. We agree on the retraining procedure and the thresholds that trigger an update in advance.
Let's discuss the task: what is needed for an accurate estimate
To come back with an estimate rather than the phrase "it depends on the data," we need inputs on the task.
The more concretely the process, data examples, and constraints are described, the more accurate the timeline and scope of work, and the fewer revisions during development — because architecture is chosen once, and redoing it is expensive. Below is a list of what most often determines both the solution and the labor intensity.
- Description of the process as it runs now. Steps, participants, where a person makes the decision and by what rules — the logic of the model and the boundaries of its application are assembled from these rules.
- Data examples. Several dozen real records together with correct answers, including disputed and erroneous ones: on them we check that the task is solvable on your data, and not only on ideal demonstration examples.
- Restrictions on data placement. Customer environment, isolated network, or processing on our side, depersonalization requirements and retention periods — this determines the choice of model and the method of delivering the result.
- Expected result and acceptance metric. What we consider a correct answer, what share of errors is acceptable at the start, who handles disputed cases manually and by what procedure.
- Integration points with current systems. Accounting systems, queues, file exchanges, web forms: where data comes from, where the result returns, in what format and how often.
- Accesses and test environment. Where we deploy the prototype, who on the customer side confirms the correctness of answers, how credentials are transferred.
After that, we agree on a plan for the first stage: data preparation and labeling, training on historical examples, metric measurement on the holdout set, and a decision on scaling.