Creating an AI product: validating the hypothesis with an MVP in 6–10 weeks

Developing an AI product MVP: where hypotheses break

AI product MVP development fails not on the model but on the framing: the team takes a hypothesis like “the model will offload operators,” immediately starts training, and defines the success metric after the fact—to fit what came out.

There is then nothing to compare the result against. We start from the opposite end: what change in the process counts as proof, and at what data volume it is even visible.

The second failure point is the quality of data for the model. The required features are spread across different systems: some in the accounting system, some in correspondence and spreadsheets, some not recorded at all.

Until a unified event history with clear labeling is assembled, quality assessment rests on a convenient subsample rather than the real stream.

The third is a demonstration that cannot be reproduced. On pre-selected examples the prototype looks convincing, but in the live stream the distribution shifts, gaps and duplicates arrive, and the model’s answer must be embedded into the processing procedure.

A pilot AI project in a limited environment is precisely needed so that this surfaces before industrial deployment.

We calculate pilot launch timelines from data readiness, not from a desired demo date: usually this is 6–10 weeks for source analysis, assembling a reference sample, the first working version, and measurement on the stream.

Testing a hypothesis with AI ends not with a demonstration but with a number that can be defended before the business and reproduced on the next slice.

What a controlled pilot delivers: measurable effect by stage

The first shift is visible even before the pilot ends: a working version on the live data stream appears a few weeks after kickoff.

The scope is deliberately narrow—one process, one source, a real export—so the result is compared with how the task was handled manually, not with an abstract promise.

We fix pilot success metrics before work begins; otherwise a dispute about the result is inevitable. The baseline is how many operations per hour a specialist processes, the error rate, and the time for data preparation.

After launch, the same values are measured on the same stream: the share of operations without a human, the processing time for one record, and accuracy on the control sample.

The main gain comes from automating routine operations and removing manual reconciliation: matching records from two systems, transferring data into the accounting environment, regular exports. This used to take a noticeable part of a shift—now a person looks only at discrepancies.

Separately, we count the eliminated class of failures: manual entry errors, lost requests, duplicates, and processing by outdated rules stop depending on a particular employee’s attention.

People remain for exceptions—control sampling and review of disputed cases instead of a full review.

By the end of the pilot, what you have is not a presentation but a comparison of the baseline with actuals.

The scaling decision is made based on numbers: if the share of operations without a human and accuracy hold at the agreed level, the scope expands to the next process, and reviewers’ corrections become labels for further training.

Work formats: closed environment, cloud, or hybrid

The deployment mode is determined by data security requirements, not development convenience: what data enters processing, whether it can leave your perimeter, and who has access to it.

This determines model placement, the point where features are computed, and what at all remains on the interface side.

The practical choice comes down to three modes—the customer’s local environment, public cloud, and a hybrid data processing scheme. They answer differently the question of where data lives and where models are computed.

We choose based on data flows: which fields are sensitive, what request volume, and how often we retrain. Without regulatory prohibitions, the cloud provides a fast start; with them, a closed environment or a hybrid with anonymization at the boundary.

We build in the boundary from the outset so that tightening requirements does not mean rewriting the product.

Stages of work: from data audit to solution handover

We work in one- to two-week iterations: each stage closes with a verifiable artifact—a data report, a solution design, or a working prototype. The discrepancy between expectations and real data surfaces in the second week, not after launch.

1. Data audit. Sources, volume, share of gaps and duplicates, labeling quality, availability of historical slices. The output is a data map and a list of what will have to be labeled anew.
2. Solution design. Where the service integrates into the current process, what the inputs and outputs are, and which metric is considered sufficient. We separately fix the boundary: what the model decides and what remains with a human.
3. Prototype assembly. The environment is assembled on real data, not a demonstration sample: data preparation, training, response service. Once a week—a demo on live examples.
4. Validation on the real stream. The pilot works with real requests, and some answers are verified manually. We calculate accuracy by class, refusal rate, and response time, and adjust the design where the metric falls short.
5. Pilot acceptance. We compare the result with the criteria from the design: share of correct triggers, share of manual corrections, behavior on rare cases. We make the expansion decision based on numbers.
6. Handover to operation. We deploy on your infrastructure, enable logging and alerts for model degradation, and train the team. We hand over the design, configurations, and update procedure.

After handover, regression tests on a fixed sample and the procedure for further training on new data remain—the solution works without our ongoing involvement.

Development artifacts: configurations, designs, procedures

The project ends not with a demonstration but with a handover: the service must evolve through your team’s efforts, without dependence on the solution’s author.

That is why we assemble the delivery so that six months later a new engineer can understand the system from documents, not from chat history. What remains with the customer:

• Data flow diagram — sources, storage locations, transformation stages, logging points, and branches where data goes to external services.
• Access policies — roles and their permissions, who can change configurations, who sees logs and personal data, and where action auditing is enabled.
• Configurations — generation parameters, confidence thresholds, request limits, moved into version-controlled files rather than settings on someone else’s server.
• AI solution documentation — architecture, dependencies, component descriptions, instructions for local launch and response quality checks.
• Test suite — test questions with reference answers and metrics, an automated run before every deployment.
• Operations procedure — what is monitored, at what deviations alerts trigger, who acts during quality degradation, and how to roll back to the previous version.
• Knowledge transfer to the team — code and incident walkthroughs with your specialists, training on real failures, and rights and accesses issued to your accounts.

Additionally, we provide a list of improvements with priorities and effort estimates—it can be used to plan development without re-entering the context.

Implementation example: request processing service

Request processing service: before implementation, an operator handled each request—read the email, transferred fields into the accounting system, selected the assignee, and sent confirmation.

At peak, the flow reached several hundred inquiries per day, the review queue grew, some requests were lost between email and Bitrix24, and the client waited hours for a response.

We separated the pipeline: the model extracts data and proposes a route, and the operator gets involved only for response quality control.

Metrics before and after

Error analysis and quality control

Once a week we analyze failures: a missing field in an attachment, an incorrectly identified request type, questionable routing. Each case goes into the reference set, the model instructions are updated, and the same error repeats less and less often.

The operator selectively checks 15% of requests; if the error rate on the sample crosses the threshold, the service switches the flow to manual mode and waits for review.

During operation, not a single inquiry was left unanswered, and automated request processing covers most of the flow.

How does a pilot differ from a demonstration stand?

A demonstration stand answers the question “is this possible in principle.” Its data is prepared in advance: duplicates are removed, formats are aligned, and representative examples are selected.

On such a sample, almost any solution looks convincing, so the stand is suitable for choosing a direction but says nothing about the system’s behavior in the working environment.

A pilot launches on real data—with gaps, heterogeneous labeling, skewed class distribution, and documents no one has checked for years.

This is where the discrepancy between quality on a representative sample and on the production stream appears: the share of correct answers on prepared examples and on live data differs, sometimes by a multiple.

Assessing whether a solution will pay off based on a stand is the same as judging a service’s reliability by screenshots of its interface.

Acceptance criteria are fixed before work starts, not adjusted to fit the result.

Usually this is a quality metric on a holdout sample, the allowable share of refusals and manual rework, the maximum response time, and, most importantly, the pre-described system behavior when confidence is low: refusal, escalation to an operator, or a “requires review” flag. Without this, the dispute about “it works or it doesn’t” turns into an exchange of opinions.

Acceptance comes with a procedure: who is responsible for quality after handover, how the model is updated when input data changes, by what signs degradation is detected, and how to roll back to the previous version.

A pilot without such a procedure ends with a working prototype that six months later no one dares to touch.

The difference in the end is simple: the stand shows that the technology exists; the pilot answers a different question—does it work on your data, with what reliability, and what will happen when the data changes.

Technical concerns: data, access, support

Concerns at this stage are usually not about model quality but about the environment around it: where data is stored, who grants access, and what happens after handover.

Below are short answers to what most often comes up during approval and affects timelines more than the algorithm itself.

What access is needed to accounting systems?

Read access is sufficient, and only to the objects needed for the task. Access is granted to a separate service account with limited rights, not to a specific employee, and is revoked with one action when work is completed. If the system does not expose data directly, we work through scheduled exports.

How is data security ensured?

Permission separation, request logging, and anonymization at the export stage close most risks. If data must not leave your environment, the solution is deployed inside your infrastructure, and only quality metrics go outside.

What happens after handover?

We hand over configurations, the data schema, deployment instructions, and the update procedure. For the first weeks after launch, we stay in touch and analyze incidents together with your team; after that—scheduled quality checks and alerts when metrics deviate from the norm.

How often should the model be retrained?

Not by the calendar but by signs: the share of manual operator corrections has increased, the distribution of incoming data has changed, or new categories have appeared.

Retraining takes hours, but the release decision is made only after validation on a holdout sample—one the model has not seen.

Let’s start with the task: discuss the data and return with a plan

To return with an effort estimate and a pilot plan, we need not a ten-page brief but a walkthrough of the task together with those who run the process.

Usually one or two conversations and access to an anonymized data slice are enough: it reveals the real structure, not a polished export from a report. Then we estimate effort by stage, identify risky areas, and propose a pilot plan with verifiable checkpoints.

What helps obtain an accurate estimate:

• A description of the process as it runs now: who makes the decision, based on what, and how much time one operation takes.
• Data in its current form: tables, exports, scans, event logs—the actual structure and its heterogeneity matter.
• Data constraints: personal data, access mode, prohibition on moving outside the environment, volume of accumulated history and its labeling.
• Project success criteria in measurable values: share of cases processed automatically, accuracy on the control sample, response time.
• Integration points: which systems the solution must exchange data with and through what.
• Who on the customer side accepts the result and how acceptance is structured.
• Timeline and regulatory constraints: when the first working result is needed and what windows exist for moving to production operation.

Describe the task in this framing—we will return with an effort estimate, a pilot plan, and a set of metrics that will show whether the solution works or not.