AI integration into the customer's existing system

Why AI integration into a system breaks at the implementation stage

A prototype on historical datasets shows quality sufficient to solve the task, but a month later in the production environment the model makes mistakes where it used to be accurate.

It is not the algorithm that breaks — it is the junction between the model and the systems that already carry operational load.

Industry reviews, including McKinsey, have documented the same thing for years: only a minority of trial launches reach industrial operation, and the reason is almost always engineering, not mathematical.

The first gap is source data quality.

For experiments, a one-off extract with smoothed reference data is made, while in the live system the same object is described differently in three places: request status comes from the accounting system, from the task queue, and from the manual edit log.

Until discrepancies are reduced to a single source of truth, any metric on the test bench remains optimistic.

The second gap is the absence of a quality evaluation loop. If quality is checked by eye on a dozen examples, the argument "did it get better or worse" is decided by the opinion of the loudest participant, not by measurement on a reference sample.

Without a fixed set of checks and rules for comparing versions, the team does not dare release updates, and the system freezes at the first build.

The third gap is manual operations around automation. An operator transfers the result to an adjacent system, a dispatcher confirms disputed triggers, an analyst reconciles reports once a week.

Automation covers only part of the step, and the rest remains manual — and that is exactly where the bulk of labor costs accumulate.

We break down how this gap is closed: we bring data to a form suitable for the model, assemble a quality evaluation loop with reference sets and rules for comparing versions, and plan a trial run without going into operation — on a real stream, but with rollback in one command.

What changes after embedding AI into workflows

Before integration, the entire flow of requests passed through an operator: parsing the email, determining the topic, responding from a template.

After, some typical scenarios are closed without a human, and the employee gets involved where the system is not confident in the answer. This is the share of automatic operations — usually 40–60% at the start, then it grows as the knowledge base is expanded.

Request processing time changes unevenly across categories, and looking at the "average" is meaningless. A typical request (request status, terms, simple calculation) is closed in seconds, together with the answer in the client's channel.

A complex one is not lost: it goes to the operator with collected context, history, and a draft response.

Answer quality control is built into the process.

The system records every decision together with the source it relied on, and we regularly compare the sample against reference answers — the share of correct hits is visible, as well as the class of errors that disappears entirely: lost requests, incorrect category, answer sent to the wrong place.

Reduction of manual operations is the main source of effect. The operator stops copying data from 1C and email and works with non-typical cases, and the load during peak hours no longer requires expanding the shift. We capture the metrics of the implementation effect on the same stream — before and after.

AI connection options: external service, closed environment, hybrid

The connection mode is chosen not by the model's capabilities, but by requirements for data and the acceptable level of control over it.

The same document parsing scenario works fine through an external service, while in another case it requires a closed environment at the customer's site — the only difference is what is allowed to leave the perimeter.

First, we fix the data requirements: which categories of information go into the request, where the copy is stored, what remains in logs and for how long.

Then we look at the limitations of the mode — dependence on an external channel, scope of maintenance, and the ability to update the model on your own.

In practice, most often they arrive at the hybrid option: the sensitive environment stays inside, typical tasks go to an external service.

We fix the final distribution with a data flow diagram and routing policy — then changing the model or provider does not break the integration.

Implementation stages: from data audit to handover to operation

Implementation begins not with the model, but with understanding what data already exists and how the solution will fit into the current environment.

The order of stages is fixed: each next one relies on the result of the previous one, and an incompatibility missed during the audit surfaces already at the moment of handover to operation.

1. Data and process audit. We examine sources, their completeness and quality, and look at where manual operations remain in the process. The output is a data map and a list of gaps that we close before the start.
2. Integration scheme. We determine the exchange mode (stream or batch processing), embedding points, access separation, and failure behavior. We agree on the scheme with your engineers before writing code.
3. Solution setup. Data preparation, validation rules, and versioning of settings — so that the result is reproducible rather than dependent on a specific run.
4. Validation on historical data. A run on past periods and comparison with the actual result. We look not at average indicators, but at rare and borderline cases — that is where the solution makes mistakes most often.
5. Acceptance of the result. Measurements by the metrics fixed before work began, on a holdout sample. Acceptance is based on numbers with a review of every discrepancy.
6. Handover to operation and team training. Operating procedures, quality monitoring, update and rollback procedures. We train your specialists and support the first period of independent work.

What we hand over: schemes, access policies, procedures

Handover is not a "folder with documents," but a working environment: the customer's engineer must be able to independently investigate an incident, change policy, and launch retraining without our involvement.

Therefore, each artifact describes not only what was done, but also how to verify it and what to do when something goes wrong.

• Data flow diagram — sources, processing points, exactly what goes into the model and where answers are written; it shows at which step the chain broke during a failure.
• Access policies — roles of services and people, what data is visible to whom, the procedure for issuing and rotating keys, the log of model requests.
• Model documentation — purpose, what data it was configured on, applicability boundaries, known limitations, and the signs by which an answer is considered correct.
• Retraining procedure — what signal triggers an update, who confirms it, what checks the new version passes before rollout, and how to return to the previous one.
• Answer quality log — a regular sample with manual labeling, the share of correct answers, the share of escalations to a human, and weekly dynamics.
• Incident procedure — who responds, within what timeframe, what actions are allowed without approval, and where to look at the history of similar cases.
• Support instructions — integration points, limits, error codes, and typical recovery scenarios on your own.

A month after launch, these materials replace correspondence with us: policy edits, failure analysis, and model degradation assessment are done inside the customer's team.

Case: how manual request processing was reduced

Before implementation, the flow of requests was handled manually: operators determined the topic and urgency themselves, looked for an answer in the knowledge base, and rewrote it for the client.

During peak hours the queue grew, and the average first response time reached 40 minutes. An error in the topic meant a repeat review — the operator returned to the correspondence and spent time twice.

What was changed

We built a labeling layer on the OpenAI language model: the request is classified by topic, urgency, and language, and entities are extracted from the text — order number, date, product, region.

Routing is built on these fields, and for typical scenarios the answer is assembled automatically from the knowledge base with a consistency check.

Scenarios with low confidence or a rare topic go to the operator, but already with a ready summary and draft response — there is no need to reread the correspondence from scratch.

Metrics after

On the established stream, 68% of requests are closed without an operator, and another portion — with one-click confirmation. First response time decreased from 40 to 4 minutes, the share of routing errors — from 12% to 3%, and repeat reviews decreased by 80%. Operators switched to complex cases instead of sorting identical requests.

Exception handling is moved to a separate loop: threshold rules, manual review of borderline requests, and a weekly sample of errors that is used to adjust labeling rules and reference data.

Operator edits go into the training sample, so quality is maintained without constant intervention by engineers, and the share of automation grows as reviewed cases accumulate.

How AI integration into a system differs from a standalone pilot

The difference is visible at the moment when the model makes a mistake. A trial run answers whether the task can be solved at all: the sample is extracted, metrics are calculated on a holdout sample, the run is performed manually and repeated until a conclusion emerges.

Industrial operation answers another question — what happens when quality on the live stream drops, and people are already using the decisions.

After a trial run, the customer is left with a report with metrics and an understanding of the quality ceiling, but not a working service.

Moving the model to production is separate engineering work: versions of the artifact and training sample are fixed, input data is validated against a schema, decisions are written to a log, and the previous version remains available for rollback.

The industrial loop adds monitoring of input data, a confidence threshold at which a request goes to a human, access separation, and an update procedure.

Post-launch support means planned retraining on accumulated labeling and analysis of discrepancies between prediction and fact. A trial run leaves knowledge and a report; AI integration into a system leaves a reproducible model rebuild process and a clear rollback point.

Technical concerns: data quality, load, rollback of changes

Engineering teams usually have the same questions: is there enough data, will the industrial loop withstand the growth in requests, and what happens if the model starts making mistakes. Below is how we close each of these concerns before work begins, rather than after an incident.

Will the existing system degrade?

We move model calls to a separate loop and keep them asynchronous: the main scenario does not wait for a response where it is not needed. We limit concurrency and timeout, and if the module is unavailable, the scenario returns to the previous logic — services operate without AI.

How do we roll back changes?

The model version, configuration, and processing rules are versioned together, and traffic is switched by a flag. The previous version remains deployed, so rollback takes minutes and does not require a rebuild. Switches are recorded in the log — you can see which version processed a specific request.

Who owns the data and who is responsible for the model?

The data remains yours: processing takes place in your environment, it does not go into other projects, and access is role-based with a request log.

After handover, the code, configurations, and documentation transfer to your team along with the model update procedure — responsibility for the result remains with you.

Let's discuss your task and come back with an estimate and work plan

To come back with an estimate and work plan rather than general words, we need access to the real environment and an understanding of the process.

To start, a conversation with your engineer and one person responsible for the process is enough — that is sufficient to fix the task boundaries and input data for the start.

• Process description: where the bottleneck is now, who accepts the result, what is considered a system error.
• Access for the audit: a test environment or database copy, an account with read permissions, access to request logs.
• Data samples for a typical period — including incomplete and contradictory records: we check quality on them before work begins.
• Integration points: source and recipient systems, which interfaces are already open, and which will need modification.
• Constraints: requirements for personal data, where information is allowed to be stored and processed.
• Current stack and the person responsible for acceptance on your side — without them, stage timelines will hang on approvals.

In response, we send a work plan: stages, their boundaries and timelines, an integration scheme with a division — what we do, what remains on your side — and a list of risks with ways to mitigate them.

The estimate is calculated based on your data, so stage timelines are met rather than shifted on the very first iteration.

Let's discuss the task in a convenient format — by email, process diagram, or call. We will come back with an estimate and work plan and clarifying questions about access and integration points.