Why image recognition in an app is slow and makes mistakes
A typical failure looks like this: the entire frame goes over the network, waits in a server queue, and returns with a response after a few seconds.
During this time the user retakes the shot or closes the screen — duplicates accumulate in the queue, and response time grows even more.
Slow image processing almost always means one thing: image recognition in a mobile app is performed far from the camera.
The frame is not prepared on the device — not resized to working size, not compressed, and obviously unsuitable shots are not discarded, so several times more data goes over the network than the model needs.
The second class of problems is recognition errors. A model trained on an averaged dataset confidently labels similar objects, while the confidence threshold is left at default and not calibrated on your data.
In poor light, at an acute angle, with glare or motion blur, labels start getting confused between neighboring classes.
The cost is visible at the next step: the share of manual photo review grows, operators sort through disputed frames instead of exceptions, and reports are built on data that cannot be trusted.
We break down how this is addressed — what to compute directly on the device, how to prepare a frame before sending, and what to use to calibrate thresholds on your own dataset.
What changes after implementation: accuracy, speed, failures
Before implementation, recognition in a mobile app works "on the whole": an object in the shot seems to be found, but on frames with glare, at an angle, or in low light, model confidence drops, and the scenario goes to manual correction.
According to industry measurements on labeled datasets, recognition accuracy for main classes rises from 80–85 to 92–97% — and holds on the production stream, not just on a demo sample.
The same data is cited in public runs MLPerf Mobile, so this is a reproducible value, not a lucky one-off experiment.
Image processing speed becomes predictable: a response on a frame arrives in 0.3–0.6 s directly on the device, without waiting for the network.
This changes the scenario itself — the employee points the camera and sees the result without leaving the workflow, while the queue of shots is processed in parallel with the main work.
The biggest shift is in the share of manual edits: where an operator previously corrected every fourth or fifth frame, only a few percent remain — they are visible in the log and can be reviewed selectively.
Entire classes of failures disappear: "did not recognize", "recognized the wrong object", "response did not arrive in time". Each of them previously cost a repeat visit or a call to the client.
We build control at input: if confidence is below threshold, the app asks to capture another frame rather than sending a questionable result to manual processing. Therefore metrics do not degrade as load grows.
Image recognition modes: on device or on server
Choosing a mode is not "better or worse" but a question of where computations are allowed and how often the model changes.
On-device processing gives a response without network, but is limited by the resources of a specific device and does not allow updating the model without releasing a new app version.
Cloud recognition removes these limits, but depends on the channel and requires transmitting frames outside.
Below are three modes as we build them into the architecture.
| Mode | When it fits | Limitations |
|---|---|---|
| On-device processing | Field, warehouse, manufacturing: connection is unstable, frames are sensitive and must not leave the device | Task class is limited by device resources; changing the model = new app release |
| Cloud recognition | Catalogs, documents, rare classes: accuracy and fast model changes matter | Stable channel required; latency depends on the network; frames leave to a third party |
| Hybrid scheme | High-volume scenarios where frames vary greatly in complexity | Routing by model confidence and cross-checking results from both sides required |
The hybrid scheme is built on a primary filter: the on-device model handles typical frames, while disputed ones go to the server together with features, not in full.
If connection is lost, frames and labels are stored in a local queue and uploaded when it is restored — offline work does not lead to data loss, and discrepancy between responses from both sides is resolved by re-checking.
Work stages: data audit, scheme, tuning, acceptance
The order of implementing image recognition is determined not by model choice, but by data quality and how much the test sample matches real user shots. First we analyze what the customer already has: shooting conditions, defect rate, labeling, class distribution.
Only after that does it make sense to discuss the scheme and timeline — otherwise high accuracy on demo images says nothing about behavior in the app.
- Data audit. We analyze the archive of shots: volume, shooting conditions, class distribution, contradictions in labeling. Output — a problem map and understanding of which examples are missing.
- Sample preparation. We remove duplicates and corrupted files, supplement rare classes, fix the held-out part of the sample that does not participate in training. It is exactly this part that gives an honest assessment of the result.
- Solution scheme. We determine what is computed on the device and what on the server, how low confidence is handled, and what the user sees when recognition fails.
- Tuning and training. We fine-tune the model on prepared data and monitor accuracy for each class separately, not only the average metric.
- Integration into the app. We embed recognition into the user scenario, measure response time and build size, check behavior under weak network and on older devices.
- Acceptance in real conditions. We run the system on fresh shots from the live stream, compare metrics with those fixed in the scheme, and analyze each error class.
- System handover. We provide configurations, data preparation scripts, metric description, and the procedure for fine-tuning on new examples so the customer's team can develop the solution independently.
Delivery package: schemes, access policies, labeling regulations
A project is delivered not when image recognition works on test frames, but when your team can change scenarios and labeling without us.
Therefore the documentation package goes together with the code: each artifact describes its link — from capture on the device to writing the result into your environment.
- Integration scheme — what is computed on the device, what goes to the server, how the app behaves under poor network, and where the failure points are.
- Contracts and formats — request and response structure, error codes, versioning: a model update does not break the released build.
- Access policies — roles of services and people, environment separation, key lifetime and rotation procedure: access is granted for a task.
- Labeling regulation — class taxonomy, rules for disputed frames, requirements for image quality, and batch acceptance procedure: a new contractor produces a comparable result.
- Processing settings — confidence threshold, frame preprocessing, failure behavior, and rate limiting under peak load.
- Metrics and measurement methodology — on which dataset we check, how we calculate accuracy and false positive rate, and at what drop retraining is triggered.
- Team instructions — how to roll out a new version, how to roll back, what to look for in logs when responses diverge.
- Handover procedure — who supports what after launch and how access and data are transferred when the contractor changes.
Case: how manual photo review in accounting was reduced
Before implementation, acceptance in TovaroUchet relied on people: a warehouse worker photographed each item, and an operator then manually compared the shot with the invoice.
A batch of 300 items took 40–50 minutes, and about every tenth shot contained an error — a mixed-up SKU, an unreadable label, an extra frame. Discrepancies surfaced only during inventory, when they had to be searched across the entire receiving history.
We restructured the process: the app captures a frame, finds the label area itself, recognizes the code and text, compares the result with the invoice, and gives the operator only those shots where model confidence is below the set threshold.
The model was fine-tuned on the customer's labeled shots — first on several thousand frames, then quarterly, adding what operators labeled during manual review. Disputed frames are not discarded: they become material for the next fine-tuning.
Metrics were taken on the held-out dataset and on the live receiving stream. Recognition accuracy for SKU from a shot rose from 91% to 98.6%, recall — to 97%, the share of shots sent to manual review fell from 100% to 7%.
Average batch acceptance time decreased from 45 to 6 minutes: the operator reviews not everything, but only conflict frames.
The main effect of automating photo review is that discrepancy is visible at the moment of acceptance, not a week later.
We calculate accuracy and recall not by average, but for each label type separately: on crumpled and glare-affected stickers the values are lower, and the confidence threshold for them is set stricter.
What to do if a third-party photo processing system is already deployed
Working recognition is not a reason to start from scratch. The first step is an inventory of three layers: the labeled dataset, the trained model, and the code that calls it. In practice, two out of three can be reused, and this changes both the scope of work and the timeline.
Labeling is the most expensive: it is collected over months and most often lost when the contractor changes.
We export data in a closed format into an open interchange format and fix a control sample — without it, the quality of the old and new model cannot be compared on the same frames.
We transfer weights via ONNX: the interchange format separates the model from the runtime, so replacing the environment does not require rebuilding the app.
| Option | When it fits | Limitations |
|---|---|---|
| Keep the model, replace the wrapper | Accuracy is satisfactory, complaints are about speed and convenience | Dependency on the contractor's closed interface remains |
| Partial migration | Labeling and weights are exportable, the app is already with users | During transition we maintain two recognition contours |
| Full replacement | Quality is insufficient, data is not given | Labeling is collected anew, time to launch increases noticeably |
The key risk when changing the contractor is not the replacement itself, but the loss of history: if dataset versions and metrics were not recorded, after transfer there is no way to prove that it did not get worse.
Therefore transfer of existing work proceeds in stages: first a run on accumulated data alongside the current solution, then gradual migration of users. Reuse of labeling here is the main resource that pays off the entire recognition migration.
Technical concerns: light, angle, data privacy
Handheld shooting is the main source of errors: a blurred frame, backlight, shadow from the hand, a tilted angle.
Before recognition, the image undergoes normalization — contrast alignment, noise reduction, bringing to a uniform scale — and on difficult frames this returns accuracy to the level of studio shooting.
Privacy is solved by architecture, not promises. By default, the frame is processed on the device itself, and only the recognition result goes to the server; if a task requires image transmission, it is de-identified, encrypted in transit, and stored for a limited term.
What to do with noisy and dark shots?
The pipeline includes local brightness correction and noise reduction. In low light, accuracy drops to 70–80%, after preprocessing it returns above 90%; the confidence threshold is configured separately so recognition does not "stay silent" on a dark frame.
How does recognition behave at a poor angle?
The model is fine-tuned on angles from your scenario — tilted shelves, racks, top-down shooting. A frame border and readiness indicator are added to the interface: a shot in an obviously unusable position is simply not taken.
How are false positives filtered out?
The scheme is two-stage: first a candidate region is selected, then it is confirmed by a classifier. The confidence threshold is configured for the cost of error — where a false positive is more expensive than a miss, disputed frames go to manual review.
Can a frame be processed without sending it to the server?
Yes: with a compact model, recognition runs entirely on the device, including offline mode. Then the shot physically does not leave the phone, and only aggregated quality statistics go to the server.
Let's discuss the task: what to consider before work starts
To come back with an estimate for a recognition project, rather than a broad range, we need inputs from both sides. The estimate rests on three pillars: which objects we recognize, how much data and in what form exists, and where we compute — on the device or on the server. You already have some answers; some we fix together and turn into a work plan.
- Scenario and object classes. What exactly we recognize — text, documents, defects, markings — and how many classes. Classes similar to each other require more data and fine-tuning; noticeably different ones can be handled by a ready model.
- Data. Whether there is an archive of photos and scans, what condition it is in, whether the sample is labeled. Diversity of shooting conditions — light, angle, background, camera quality — directly determines the scope of work on preparing and checking the dataset.
- Where to compute. On the device, the result is available without connection and the frame does not leave the phone; on the server, accuracy is higher and logic is easier to update. The choice sets the composition of the client side and channel requirements.
- Quality requirements. What share of correct detections we consider sufficient and what is more critical: a miss or a false positive. Without this, thresholds cannot be configured, nor the result accepted.
- Where to send the result. Accounting system, warehouse, ticket database: in what form and on what event the recognized result goes out, what access rights and logging exist.
- Constraints. Supported platforms and OS versions, offline mode, acceptable response time, requirements for storing personal data in the frame.
- Acceptance and timeline. On which set of examples we check, who approves stages, whether there is an external release date.
Describe the task in free form, even if some points are still open. We will come back with an estimate, work plan, and stage order, and fill in the missing inputs during the review.