AI Agent Development: Why a Prototype Does Not Reach Production
An agent pilot in production almost always starts the same way: on prepared questions the agent answers smoothly, and it seems that only one step remains before automating request processing.
In the real process, it gets confused in reference data — picks an outdated SKU, mixes up two legal entities, inserts a condition from someone else's contract. The answer has to be checked and corrected manually, and no time savings remain.
What breaks here is not the model.
What breaks is the link “model — data — access rights”: the agent’s integration with accounting systems is reduced to a “give me everything” request, there is no single key between reference data sources, and the service account has been granted full rights.
The agent receives conflicting records, selects the first one in the output — and returns an answer that the operator cannot accept without checking against the source.
The quality of an agent’s answers is determined by what it sees at the input.
Without rights-based filtering, it either answers “no data” or shows extra information; without deduplication of reference data, it selects an outdated record; without preserving context, it loses the thread on the third clarification.
Without a call log, the solution cannot be analyzed after the fact: it is unclear which system the record came from and why the agent chose that one. None of these failures is visible in a demo — they appear only on a live flow of requests.
We examine exactly where the link breaks during AI agent development and what changes in the architecture so that the pilot reaches operation rather than remaining a nice-looking prototype.
What Changes in Processes After Agent Implementation
The effect of the agent is measured not in a demo but in the department’s operational metrics. Before launch, we record baseline values for five metrics; after, we compare: the difference is the result of implementation.
The agent runs on top of YandexGPT models, so parsing Russian-language emails and requests happens without exporting data outside the company’s perimeter.
- Share of automated operations — the agent handles typical scenarios (reconciliation of details, parsing an incoming email, preparing an export according to the procedure) end to end, without an operator. On a limited perimeter this is usually 60–75% of the volume; the remainder comes to a human with context already assembled.
- Request processing time — classification and the initial response take seconds instead of hours waiting in a queue; night and weekend requests no longer pile up until morning.
- Data entry errors — the agent transfers data from the document into the accounting system directly, without manual retyping, and a whole class of defects disappears: typos in details and amounts that land in the wrong field.
- Department throughput — grows without expanding headcount: the same team handles more requests on peak days, and the queue is cleared within the shift.
- Onboarding of new employees — the agent suggests the next step according to the procedure and pulls up the needed customer card, so a newcomer reaches the norm in weeks rather than months.
- Process cost — decreases not by reducing people but because routine work stops requiring manual hours: storage and maintenance costs for the perimeter do not grow with volume.
Important caveat: we take metrics on the same set of requests; otherwise the comparison proves nothing.
Agent Implementation Options and Their Limitations
Agent implementation options differ not by the model’s “intelligence” but by which systems it is connected to and who is responsible for its behavior. A ready-made agent builder provides a fast start; a custom perimeter provides full control over logic and data. The choice comes down to the number of integrations and the required autonomy.
| Option | When it fits | Solution limitations |
|---|---|---|
| Built-in assistant in the interface | Need answers on procedures and data without changing processes | No access to internal systems, does not perform actions, limited by screen context |
| Agent with access to internal systems | Need to read and change data in accounting systems | Requires access separation, an action log, and handling failures of third-party services |
| Assembly on a ready-made platform | Scenario is typical, launch speed matters | Ceiling on scenarios, dependence on the platform vendor, non-standard rules are difficult to embed |
| Fully custom perimeter | Unique logic, own data storage rules | Longer time to put into operation, requires ongoing support and model maintenance |
The main limitation of any option lies not in the model but in access: without access separation, the agent is either useless or dangerous for data. That is why we first describe scenarios and write points, and only then choose the implementation option.
Engineering Work Stages: From Audit to Handover
AI agent development starts not with code but with analyzing how the process is structured now.
We look at which systems participate in request processing, where data is stored in a machine-friendly form and where it lives in emails and spreadsheets, who makes the final decision, and what counts as a correct answer.
Based on the process audit, we record which steps the agent takes over entirely and which it leaves to a human with a ready-made suggestion.
- Process and data audit. We go through each processing scenario, noting sources, request frequency, and typical branches. The output is a process map with bottlenecks where manual work consumes the most time.
- Integration scheme. We determine which systems the agent accesses for reading and writing, how data is transferred, and what happens when an external service is unavailable. The integration scheme is agreed before assembly begins — edits at this stage are cheaper than reworking the logic.
- Setting up rights and access. The agent receives the minimum necessary set of rights, data-changing actions go through confirmation, and all requests are written to the log. This makes it possible to separate what the agent does itself from what it only proposes.
- Scenario assembly. We assemble the logic step by step: parsing the incoming request, selecting the data source, forming the response, handling failures. Each scenario ends with a clear result — a card, an email, or a record in the system.
- Testing on historical data. We run the agent on accumulated requests and compare the answers with what employees did. We analyze discrepancies: some are resolved by refining the scenario, others by clarifying the rules.
- Pilot operation. We launch it on a limited flow in parallel with the current process, compare the result, and keep manual control on disputed cases. This way, a failure in one scenario does not affect the entire volume of requests.
- Handover of the solution to the customer. We hand over the configured system, documentation on scenarios and logs, and train the team to make edits and analyze incidents. After that, maintenance is handled by the customer or with our support.
Documentation, Schemes, and Maintenance Procedures
An agent without documentation quickly becomes a black box: the logic is reconstructed from logs and guesses, and every edit requires the same engineer who wrote it.
That is why our AI agent development ends not with launch but with the handover of a set of materials that your support team can use to work independently. Below is what exactly remains in your hands.
- Agent documentation — data flow diagram: which systems provide data, where the agent takes the knowledge base from, where it writes the result, which external calls may fail to respond, and how failures are handled.
- Description of scenarios and prompts — for each request type: what triggers the agent, which steps it goes through, where its authority ends, and under what condition the dialogue goes to an operator.
- Access matrix — the list of accounts and agent rights in each system, explicit prohibitions, the access owner, and the procedure for revoking it when an employee changes.
- Set of test scenarios — labeled requests with reference answers: a run before each update shows that the new version has not broken previously working branches.
- Support instructions — what to do in typical failures: silence from an external system, an empty or irrelevant answer, incorrect routing, manual transfer of the dialogue to a human.
- Maintenance procedure — frequency of quality checks, procedure for editing the prompt, rules for rolling back to the previous version, responsible parties, and response time to a request.
- Change log — version, essence of the edit, reason, and which tests it was checked against.
The set is not a report for the shelf but a working tool: it is updated together with the agent; otherwise, in a couple of months the descriptions diverge from the system’s real behavior and people stop trusting them.
Case Study: How an Agent Offloaded Incoming Request Processing
Where the Process Was Stuck
The request processing service was working at its limit: on peak days, up to 400 unprocessed requests were hanging in the queue, and the client received the first response after 6–9 hours.
More than half of the requests were typical — request status, access extension, a set of closing documents.
The operator manually collected data from three systems, transferred it into the response, and still returned every fifth request for rework: the text lacked context, and the client wrote again.
What Exactly Was Done
The agent was embedded not “above” the process but inside it. At the input, it parses the request text, determines the request type, and extracts entities — contract number, counterparty, period.
Then, according to routing rules, it accesses the Single Window and assembles a response from verified templates; if recognition confidence is below a set threshold, the request goes to an operator together with a ready draft and the client’s history.
The second layer is quality control. Every automatically sent response is marked in the log, and returns for rework are analyzed separately: discrepancies become new rules and replenish the set of examples.
Rules are versioned, any agent action is logged, and an erroneous response is rolled back and switched to manual mode.
Metrics After Three Months
The share of automatic request closures reached 43% of the entire incoming flow. First response time decreased from 6–9 hours to 4 minutes during working hours and to 15 minutes at night.
The share of returns for rework fell from 21% to 6% — mainly because the operator receives the request with context already assembled. The load on the shift decreased by about a third, and the same people began handling disputed cases instead of copying data between windows.
What to Do If an Agent Prototype Has Already Been Built on a Platform
A built prototype is an asset, not a draft: a review is usually faster than building from scratch and shows which scenarios are confirmed on real data and which rest on demonstration examples.
Hence the choice between two options — refinement instead of rewriting, or moving the logic into a managed perimeter.
A prototype review is running the finished agent prototype on real requests from a past period, not on prepared examples.
We look at the share of answers accepted by an employee without edits, behavior on empty, contradictory, and overly long input, and the number of model calls per task: the latter determines how the agent will behave as load grows.
Next, we calculate technical debt. A typical set: prompts and rules live in code without versioning, dialogue state is stored in process memory and is lost on restart, there is no call log or separation of rights for tools, and rollback is possible only by fully shutting down. Each item is a separate class of operational failures, not cosmetics.
Refinement instead of rewriting is justified when scenarios are confirmed and the debt is local and can be removed on top of the existing code.
Moving to a managed perimeter is needed if the prototype was built on a builder platform such as n8n and has hit its limits: no fine-grained error handling, difficult to separate environments, impossible to assemble a regression set. There is one criterion — what is easier to keep in working condition.
Bringing the prototype into operation adds a log and step tracing, call rate limits, regression checks on real cases, separate environments, and a rollback plan.
We enable it in stages: first one process with manual result verification, then, as the share of accepted answers grows, further.
Common Technical Concerns Before Starting a Project
Concerns at the outset are almost always the same — we address them with architecture, not promises.
Where does our data go?
The agent runs in your perimeter, and the model is called through a logging gateway. Only the context needed for the step goes into the request; personal fields are masked at the input, and access to logs is separated by roles.
What rights does the agent receive?
A separate service account with the minimum necessary rights: reading reference data, creating a draft, sending for approval. Modification and deletion remain with a human — confirmation is built into the scenario as a mandatory step.
How do you check answer quality?
Before launch, we assemble reference scenarios with the expected result and run them on every edit to instructions or model change. The agent does not give an answer without support from a document — if context is insufficient, the dialogue goes to an operator.
Won’t we get locked into one model vendor?
The scenario logic is separated from the invocation layer: models are connected as replaceable modules with a single interface. Replacing a vendor means editing the configuration and rerunning the reference set, while the instructions and knowledge base remain yours.
Who is responsible for the result after handover?
We hand over scenarios, instructions, and observability: call logs, quality metrics, and the procedure for rolling back to the previous version.
Your engineers are trained on your own perimeter; the warranty period covers implementation defects, and after that, support follows the maintenance procedure.
What to Prepare So We Can Return with an Estimate and Plan
To keep the discussion concrete and the estimate based on facts rather than guesses, at the start it is enough to describe the process in terms of the task.
We handle the model choice, integration method, and infrastructure ourselves — we need the context from you that determines the scope of work.
- Process boundaries: what event it starts with, what result it ends with, and who participates in it.
- Data source systems and the method of access to them — API, file export, database, read-only access.
- Examples of real input data: 10–20 anonymized requests or documents with the expected result — they determine the quality metric.
- What the agent does itself and what a human confirms: where an operator is mandatory, where automatic action is acceptable.
- Constraints: personal data, storage and logging requirements, deployment perimeter — your own infrastructure or cloud.
- Volume: how many requests per day, peak load, number of scenarios and document types.
- Acceptance criterion: share of correct answers, processing time, share of tasks removed from the operator.
With this set, we return with a project estimate: a stage plan and the scope of work for each — data collection, example labeling, agent setup, validation on a holdout sample, implementation, and handover.
We fix the interaction procedure in advance: where we show an interim result, who on your side accepts it, and at which stage the support team joins.
