VLM development: where the bottleneck is in image analysis
A typical failure starts with architecture: image recognition and text parsing are split into two independent steps, and document analysis is assembled from their answers.
The classifier looks at pixels and does not know what is written in the adjacent field, while the text module does not see which fragment the line belongs to.
On a scan of an invoice or form, meaning emerges only in combination — a stamp, signature, number, and table mean exactly what they mean together, while separately they give contradictory answers.
Next, the share of false positives grows. The model confidently marks an empty form as filled, confuses similar details, reacts to glare from lamination or a watermark. Each such decision must be reviewed by a human, and along with the flow, the volume of manual checks grows.
When spot-checking turns into a mandatory stage, the whole point of automation is lost: the processing time for one document increases, and the queue accumulates faster than it can be cleared.
At the same time, real defects — overlapping text, low contrast, non-standard layout — go unnoticed because the reviewer's attention is occupied by false ones.
A multimodal model removes this fork: image and text enter it as a single input, and the decision is made based on joint context, not on two inconsistent answers. Below we discuss how this is handled in a production pipeline.
What VLM development delivers: measurable effect
The difference between a demo and a working system is visible only in measurements. We fix three values before the start: recognition accuracy on a reference sample of your real documents, processing time per unit, and the share of cases that go to manual review.
Then any change — model fine-tuning, new preprocessing, editing parsing rules — is accepted or rejected based on these numbers, not on the impression from a demo.
Model fine-tuning provides the main gain where the general model makes mistakes on your material: non-standard forms, handwritten fields, industry terminology.
We collect annotations from already accumulated data, fine-tune on them, and validate on a held-out part of the sample — recognition accuracy usually rises from 80–85% to 93–97%, and more noticeably on rare classes than on frequent ones.
Each fine-tuning run is compared with the previous version so as not to lose what has already been achieved.
The effect is fastest to show in manual review. When the system sends a human only disputed cases — low model confidence, conflict between fields, unreadable fragment — the share of manual review drops from the full flow to 5–15%. The freed time goes to complex cases instead of retyping typical ones.
A separate result is the eliminated failure class. Previously one unfamiliar format broke the parsing of an entire batch; now it is isolated: the document is flagged and goes to a review queue, while the rest are processed further.
Processing time for one document is measured in seconds instead of minutes of manual work, because recognition, field extraction, and checks run in a single pass.
In projects, we rely on open multimodal models such as Qwen-VL, fine-tuning them on the customer's data. This removes vendor lock-in: the system can be deployed in your environment and the model can be changed without rewriting the logic.
Comparison of operating modes: cloud, local environment, hybrid
Operating mode is not only about where models are computed. It determines how images and text enter the environment, who is responsible for updating weights, and what happens when the request flow grows several times over.
The choice between cloud environment, local environment, and hybrid scheme is dictated by two constraints: data confidentiality and channel capacity between the source and the model.
Below is a comparison along three axes: where data is processed, what load the scheme withstands, and how much attention maintenance requires. The limitations are stated honestly — each mode has scenarios in which it loses to the other two.
| Mode | When it fits | Limitations |
|---|---|---|
| Cloud environment | Uneven flow with peaks; fast start without purchasing equipment | Data leaves the perimeter; latency depends on the channel |
| Local environment | Data cannot be taken outside the perimeter; predictable load; operation without an external network | Weight updates and equipment maintenance are on your side; performance ceiling is set by hardware |
| Hybrid scheme | Sensitive data is processed on site, heavy tasks go outside | Two environments: model synchronization and request routing require a regulation |
In practice, the boundary is drawn by data, not by convenience. If the input stream contains faces, documents, or production line footage, a local environment immediately addresses confidentiality.
A hybrid scheme wins when preprocessing and cutting out sensitive areas happen on site, while scene recognition and text parsing happen where it is easier to scale capacity.
We fix the mode before work begins: deployment scheme, hardware requirements, and model update procedure depend on it. Describe your data flow and constraints — we will come back with an estimate and a plan.
How we conduct implementation: stages of engineering work
Implementation of a VLM system begins not with model selection, but with analyzing what data the customer already has and what is considered a correct answer.
At this step we fix the input formats, flow volume, and quality metric — without it, the pilot result can neither be accepted nor disputed. Then follows a sequence of stages, each building on the previous one.
- Data audit. We look at what is actually stored in repositories and email: scans, photographs, exports from accounting systems. We check resolution, duplicates, corrupted files, and the share of examples where the answer is ambiguous even for a human.
- Solution design and acceptance criteria. We determine how the system is integrated — batch on a schedule or in real time — and agree on what share of correct answers counts as success and what level of false positives is acceptable.
- Data annotation. We write annotation rules, annotate a control set, and always set aside a portion of examples that the model will not see until acceptance.
- Pilot launch. We bring up the system on a limited flow and compare its result with the current manual process on the same files — this shows the real gain, not a laboratory one.
- Model validation. We break down errors by type, retrain on disputed cases, and check robustness to poor scans and non-standard angles.
- Solution handover. We deliver documentation, operator instructions, and configured monitoring so that the customer's team can maintain the system without us.
Each stage ends with an artifact: annotation rules, a quality report, an integration scheme — these show where we are and what remains to be done.
What we hand over: configurations, schemes, regulations
A ready solution without handed-over artifacts lives only until the first data change or engineer turnover.
That is why, together with the model, we deliver a set of documents and configurations that your team can use to reproduce the build, explain the system's answers, and change individual components without rebuilding everything from scratch.
The set resides in your repository and is updated together with the code: a change in annotation or access rules goes through the same review as a change in the solution itself.
- Working configurations for training and inference: run parameters, data versions, reproducible training conditions, and launch scripts — the run can be repeated and yields comparable quality.
- Annotation regulations: instructions for annotators with rules for disputed cases, an inter-annotator agreement metric, and a procedure for reviewing examples where opinions diverged.
- Integration scheme: data flows from source to model response, connection points via APIs and message queues, request and response contracts, and response time limits.
- Access policies: roles and rights to change configurations, read logs, and export data, environment separation, and procedures for storing credentials and keys.
- Solution documentation: a map of components and dependencies, deployment and update instructions, and a list of used datasets with their versions.
- Quality monitoring: a metrics dashboard — failure rate, answer distribution, model confidence on the live stream — alert rules, and procedures for degradation.
- Maintenance regulations: how to fine-tune the model on new data, how many examples are needed for that, and how to roll back to the previous version if quality drops.
- Decision log: why these thresholds and architecture were chosen, which hypotheses were tested and what did not work — the next engineer does not have to go through this path again.
We verify each item: if an operation cannot be performed using the document without its author, we revise it. Handover is closed by acceptance, where your team deploys the solution itself and makes the first change.
Case study: defect control on a production line
On the flexible packaging line of «TekhnoPolimer», defects were caught by three inspectors at the output: micro-tears, print color unevenness, fold misalignment.
Spot checks once every twenty rolls gave 6–8% missed defects — they surfaced at the customer's site, not in the workshop.
Disputed frames were reviewed manually: 3–4 minutes per frame, up to a third of the captured material went to rechecking, and the shift ended with a report that no one had time to read.
We installed cameras at the output of each line and a VLM analysis module: the model looks simultaneously at the image and the text regulation, so a new defect type is described in words, rather than by re-annotating thousands of frames.
Escalation rules were set by class — with borderline confidence the image goes to an operator, while the rest falls into the shift report with reference to the roll and length.
The share of missed defects decreased to 0.5%, and reviewing a disputed case takes about 20 seconds instead of 3–4 minutes. Quality control stopped being sample-based: coverage is continuous, while a shift has 60–80 operator requests instead of manually viewing the entire stream.
The workload was redistributed — three inspectors per line were replaced by one who handles escalation review and class fine-tuning. A new defect class is introduced in a day: text description, a dozen examples, validation on the archive of captured material.
How does fine-tuning differ from few-shot prompting?
Both approaches solve the same problem — getting the model to answer in the required format — but they act at different levels. Few-shot prompting changes only the request: the model receives several annotated samples and repeats their logic. Fine-tuning changes the weights — the rule is fixed in the model itself, not in every request.
Few-shot prompting is enough when there are ten to fifteen classes: extracting fields from a scan, determining document type, marking a defect in a photo. It works from day one, and a fix is replacing an example, without retraining.
The weak spot is rare and visually heterogeneous cases: the request grows, response time too, and the rule remains unstable.
Fine-tuning is needed when there are dozens of classes or the annotation is domain-specific — say, defects in images. The adapter is trained on several hundred annotated pairs, a small part of the weights is updated, so the cycle takes hours.
The scheme with adapters on top of open models such as LLaVA is a typical path: the request is reduced to an instruction without examples, and the response time per image drops.
Choosing the method is a measurement, not a matter of taste. We collect 30–50 annotated examples, test few-shot prompting on a held-out part, and look at the share of correct answers on rare classes. If quality hits a ceiling, we train an adapter and compare the same metrics on the same set.
Fine-tuning has a limitation — maintenance: a registry of adapter versions, a separate validation set, and a rollback plan when annotation changes.
Few-shot prompting has a dependence on example selection: when the set changes, answers change, and reproducibility has to be ensured manually.
Technical concerns: addressing them point by point
The customer's engineering teams ask similar questions before the start: will there be enough data, where will images go, will the solution coexist with an already deployed environment, and who is responsible for quality after launch. Below are short answers without marketing.
What will happen to rare classes if there are few of them in the annotation?
We calculate metrics separately for each class, not as an average over the sample — otherwise a rare class “drowns” in the overall figure. For it, we add augmentation, weight errors, and collect borderline examples; the report shows exactly where the model is confused.
Can data be kept from leaving the environment?
Yes. Training and inference are deployed inside your environment, annotation is performed by a limited circle of people with role-based access, and source data does not leave the storage. Only anonymized quality metrics go outside.
We already have an environment deployed — will we have to redo everything?
No. The solution is integrated alongside: its own storages, its own queues, existing authorization and access separation. Only the call points change, not the working systems.
How do we know the solution is ready for implementation?
We run an evaluation on a held-out set and compare with the current process, not with an abstract threshold. If quality is no worse, we enable it on part of the flow and manually verify the result.
What happens after launch?
We monitor data drift and quality by class, and review disputed cases. Retraining happens on schedule or immediately when a metric falls below the agreed threshold.
Let's discuss the task: where work begins
The first conversation is about the task and data, not about models. We need to understand what is already available, what result is considered correct, and where it integrates into current processes.
The absence of ready annotations does not block the start: we will assemble a trial set and check whether the task can be solved with the available data before building the entire system.
What helps to quickly arrive at an estimate:
- 20–30 real examples of input data — they show the diversity of scenarios, not an idealized picture.
- Formulation of the target result: what is considered a correct answer and what is an error.
- Integration points: where data comes from and where the result goes.
- Who on the customer's side resolves disputed cases and approves acceptance.
- Confidentiality constraints and data storage rules.
- Expected load: how many requests and how quickly a response is needed.
- Readiness criterion by which you will judge the result.
We come back not with a single number, but with a staged work plan: data preparation, baseline version, quality measurement, fine-tuning on your examples, integration, maintenance.
The plan includes the team composition with roles and timelines for each stage — it shows where scope can be cut without losing the result and where it cannot. Describe the task in any convenient form — we will come back with an estimate and a plan.
