AI consulting services: why a pilot never reaches release
The typical picture: a model is trained on an export from storage in a separate environment, a quality metric is calculated on a holdout sample, the result is shown to management—and then nothing happens. The pilot does not reach production operation because of a weak algorithm.
The reason is the gap between the prototype and the working system: what worked on a prepared data slice does not transfer to the production environment automatically.
In a separate environment, the data is already cleaned: duplicates removed, outliers filtered, and the field schema stable. In production systems, the same stream arrives with delays, gaps, and structural changes—quality drops on live data, and there is no way to explain it.
A data and infrastructure maturity assessment before the start would show which sources are missing and where feature re-engineering will be required.
The second gap is the metric. Model quality is measured by accuracy on a test sample, but it is not tied to the indicator the customer is responsible for: the share of manual edits, request processing time, or the number of missed cases.
Without such linkage, the pilot can be neither accepted nor rejected—it remains a demonstration of capabilities.
The third is integrations. Deploying models into production systems requires artifact versioning, data drift monitoring, prediction logging, access control, and rollback to a previous version. This is engineering work, not model retraining.
AI consulting services begin with an assessment of data and infrastructure maturity, not with algorithm selection. We examine how this gap is closed at each stage.
Impact of implementation: model deployment speed and failure rate
We calculate impact using four metrics that we measure before the start, during the pilot, and one month after release. If measurement is not fixed in advance, there will be nothing to confirm improvement later—only impressions remain.
Time to deploy model into production — weeks, not quarters
The first end-to-end path—request intake, model invocation, response in the production interface—we bring up on a real data flow, not synthetic data.
We check labeling quality and integration with systems of record on live requests, so time to deploy the model into production is reduced to a few weeks: most time goes to data and acceptance, not rewriting integrations.
Automated response rate
We calculate the share of requests closed without an operator and, separately, accuracy on those routed to auto-close.
Routing requests by type and complexity between models, including OpenAI, together with a confidence threshold increases the automated response rate without a drop in quality: disputed cases go to a human, not to the client.
Request processing time and manual operations
For typical requests, request processing time drops from hours to seconds—the model answers immediately, and an operator gets involved only for exceptions.
The reduction in manual operations is visible in shift workload: the same team handles a larger flow without expanding headcount.
The class of failures that goes away
The class of failures related to human error goes away: forgotten replies, lost requests, different answers to the same question.
Requests and responses are saved in logs, so an incident is investigated based on facts, not recollections—and is reproduced in testing, not with the client.
Work formats: pilot, product, integration into the customer's environment
The three formats differ not in scope of work but in what you risk: hypothesis validation runs on a narrow scenario, the product is designed from the start for multiple teams and load growth, while embedding into the customer's environment makes the model part of an already operating process.
The choice is determined by the state of the data, not by desire. If there are few labeled examples, hypothesis validation on a limited environment is cheaper than an architecture mistake.
If the scenario is validated and different departments use the service, multi-tenancy, data isolation, and quotas are needed from day one. When embedding into an existing system, restrictions on access, logging, and rollback are added.
| Format | When it fits | Limitations |
|---|---|---|
| Hypothesis validation on a limited environment | Data exists for one scenario; achievable quality needs to be assessed | Narrow set of scenarios, manual verification of results, no headroom for load |
| Separate product under load | Scenario is validated, multiple teams, data isolation needed | Its own update procedures, quality monitoring, access control |
| Embedding model into an existing system | Process already works; model complements it in familiar interfaces | Dependence on the pace of changes in adjacent systems, access and logging |
| Post-launch support | Quality needs to be maintained as data accumulates | A process owner and regular metric reconciliation are needed |
Limitations of formats
Before the start, we fix target quality metrics, data composition, rollback requirements, and acceptable response latency—they determine which format will deliver results without rework. We choose the support mode to match: continuous or on request.
Implementation stages: from data audit to handover to support
The sequence of work almost does not depend on the industry: first we determine what actually exists in the data and which number we consider the result, then we build and test on what the model has not yet seen.
Each stage ends with an artifact that can be opened and challenged—before weeks of work go into the solution.
- Data audit and metric definition. We examine which events and labels actually exist, where there are gaps, duplicates, and breaks, and whether there is enough history. We fix the goal as a number—share of correct answers, accuracy on a control sample, response time—otherwise there is nothing to compare acceptance against.
- Solution architecture design. The environment diagram: where data is stored, what is computed in batch, what is required in real time, how integrations and access rights are structured. We separately describe system behavior when the model is unsure of an answer.
- Model setup and training. Selecting the class of models for the task, preparing features, additional labeling of missing examples. We compare options on a holdout sample and fix the configuration together with the data version.
- Acceptance checks on historical data. A run on a period the model has not seen, and reconciliation with the current process. We look not only at average quality but also at behavior on rare classes and boundaries.
- Pilot on live data. A limited environment, monitoring divergence from historical results, availability, and response time. Adjustments based on results go into the same version, not scattered.
- Handover of the solution to support. Retraining procedures, monitoring of metrics and data drift, rollback order, and instructions for the customer's team. We accept it together with the responsible parties, rather than "delivering it by act."
Project artifacts: diagrams, access policies, procedures
Delivering a solution means handing over not only a working service but also a package by which it can be operated, retrained, and handed over to another team. Below is what remains with the customer in the repository and internal knowledge base after acceptance.
- Data flow diagrams: where records come from, where they are de-identified, what exactly goes into the model, where results and logs are stored—with marks for points sensitive to leakage.
- Environment configurations: parameters of development and production environments, environment variables, load limits, and maximum response waiting time. The environment can be brought up from scratch according to the instructions, without the solution's author.
- Access policies and role separation: who sees source data, who can change model settings, who only reads reports; key rotation rules and contractor access procedures.
- Model update procedure: by which metrics a new version is recognized as better than the current one, who makes that decision, and how rollback is performed when quality drops.
- Operational documentation: typical failures and actions of the on-call shift, alert thresholds, owners responsible for the area, escalation procedure.
- A set of control queries with reference answers: a run before each update shows whether scenarios that worked before have broken.
- Engineering decision log: why this architecture was chosen and which options were rejected. This is the context that is usually lost along with the team.
The package remains with you in working form: any engineer familiar with the domain can extend it, not only the authors of the solution.
Case study: automated response rate and request processing time
What it was like before
Support received requests from four channels, and all of them went into a common queue: the operator determined the topic and urgency when opening the ticket.
Classification depended on the shift, so typical questions regularly went to the second line, and the median first response time stayed around six hours. Every fifth client returned with the same question.
What was changed in the environment
The channels were consolidated into a single intake layer on top of Zendesk, which was already in use—each request got one ticket and shared context.
At the input, we placed topic and urgency detection, an auto-response from the knowledge base with a confidence threshold, and escalation rules: above the threshold, the answer goes to the client; below it, the request goes to the right queue along with a draft for the operator.
Every decision is logged with its source: the operator sees which articles the answer was assembled from, and their edits go into the labeling for retraining.
What happened in the live flow
The share of requests closed without an operator reached 38%—while auto-responses do not go beyond confirmed topics. The median first response time decreased from six hours to 20 minutes, and the 95th percentile from 19 hours to 1.5 hours.
First-time routing to the right queue increased from 61% to 94%, repeat requests on the same topic decreased from 18% to 7%, manual responses on the first line nearly halved, and freed-up engineers switched to handling complex cases.
The auto-response threshold, topic list, and escalation rules are stored in configuration and can be changed without rebuilding. Rollback is done by removing a flag: the flow proceeds as before, while the accumulated labeling remains for the next launch.
What to do if AI is already implemented in a piecemeal way and does not deliver impact
Most often the picture is like this: sales has its own email assistant, support has a request classifier, and analysts have document recognition. Each solution was adopted as a separate task, with its own data and its own metrics. Locally there is benefit, but there is no cumulative effect.
The reason is not model quality but fragmentation.
Disparate models in a company duplicate the preparation of the same data, the output of one step does not become the input for another, and there is no way to check what actually works—a common picture of quality and failures simply does not exist.
We start with an inventory of solutions: which tasks are covered, on what data, who the owner is, how the solution is invoked, and what it shows in terms of quality. The output is a map showing duplicates, dead ends, and what is already ready for reuse.
Next we assemble a unified environment instead of a set of scripts: shared data access, unified invocation rules, logging, and quality control at each step.
Reuse of existing work is key here—a working classifier is not rewritten but becomes a separate step in the environment and starts working on adjacent tasks.
We build the plan for moving beyond piecemeal implementations based on impact: first we connect what gives the greatest return, then we migrate the rest, and we decommission solutions that did not justify themselves.
Working solutions are not stopped in the process—the transition is staged so that current processes do not degrade.
Frequent questions: data, access, security, support
Questions before the start rarely concern the model—more often they are about data and who is responsible for the result after launch. Below is what technical leaders clarify first.
Data is fragmented and labeled inconsistently—where do we start?
With profiling: we count duplicates, gaps, and discrepancies in reference data between systems. Then we fix labeling rules, version the samples, and assemble a reference set on which we measure quality.
How is access to data separated?
Roles at the level of tables and individual fields, de-identification of sensitive values, access log. We conduct training and debugging on a copy without direct identifiers; access to the production environment is named and time-limited.
How does the solution behave during a failure?
Failed calls go to a queue and are retried with increasing delay; operations are idempotent—a retry does not create duplicates.
If a response cannot be obtained, the request goes to manual review, and the solution returns a result according to a fallback rule rather than failing entirely.
What is included in post-launch support?
We hand over procedures, quality control dashboards, and incident review order, and train the team. After that—data drift monitoring, scheduled updates, and retraining: without this, quality noticeably degrades within a few months.
Let's discuss the task and come back with an estimate and work plan
For the review of the current environment to rely on facts rather than general words, a few inputs are enough to start.
There is no need to prepare a technical specification: what is needed to start is a description of the task and examples of real data; we will assemble the rest together during the discussion.
What to send:
- the task in your own words: how the process works now and what result is needed at the output;
- 2-3 data examples—an export, documents, a database fragment, de-identified if necessary;
- a list of systems with which the solution must exchange data, and who is responsible for access;
- constraints: deadlines, requirements for storage and access separation, who makes decisions;
- what you have already tried and why the result was unsatisfactory;
- contact of the person who leads the area and can clarify details during the review.
We will ask some questions ourselves: whether there is enough data for the declared quality, who verifies the result on your side, how the solution fits into current procedures, and what happens if the model fails.
In response, we will come back with an estimate and work plan—stages, deliverables for each, dependencies, risks, and points where your review is needed.
The next step is simple: describe the task in free form or send what you already have. We will discuss it substantively, come back with an estimate and work plan, and propose a format for the first review of your environment.