Text recognition in a mobile app: where accuracy is lost
Recognition accuracy in a mobile app rarely breaks on a perfect scan — it collapses on how the document is actually captured.
A glare from a lamp on a glossy receipt eats up half a line during binarization, a hand shadow turns an area “gray,” and the text detection layer stops seeing blocks.
A parser that achieved 98% on a test set drops to 80–85% on a live field stream — this is no longer cosmetic, but manual correction by an operator. Every such document returns to the queue, and recognition accuracy stops being a technical metric — it becomes a process bottleneck.
Geometry adds its own issues: a 10–15 degree tilt, creases at the fold, perspective from shooting at an angle. Linear correction cannot handle such cases — you need rectifying deformation along the page boundaries, otherwise lines “float” and characters merge.
Complex forms are a separate class of errors: fields with small print, table grids, mixed layouts in one line, where Cyrillic sits next to Latin and digits inside a word.
The parser confidently reads the header and makes a mistake specifically in the policy number, which is exactly what the accounting system needs.
The cost is predictable: the share of manual input grows, processing delays appear, and some documents go for re-capture. Next, we break down how the document capture pipeline and the recognition accuracy control that addresses this are built.
What changes after implementation: speed, accuracy, autonomy
Automation of manual input is measured not by “convenience” but by the share of documents that enter accounting without operator corrections.
Before implementation, this share usually stays at 55-70%: some details are not read, some are read incorrectly — and are still retyped by hand.
After configuring recognition for specific document types and field markup, the figure reaches 90-95% and continues to grow: operator corrections return to the system as feedback.
At the same time, images do not go to a third-party server — recognition runs on the device, on the proven Tesseract engine.
Data entry speed changes qualitatively. Instead of the chain “captured → sent → waited for a response → verified,” the user sees filled fields 1-2 seconds after capture, even before writing to the accounting system.
On a batch of several dozen invoices, this is the difference between a pause at every document and continuous work throughout a shift.
Working without connectivity stops being a limitation: the model resides inside the app, so recognition is available in a basement, in the field, and on a plane.
The result is placed in a local queue and goes to the system when the network appears — without losing documents and without duplicates on re-send.
The reduction in the share of corrections does not show up immediately, but as data accumulates: typical errors — similar characters, skew, glare — are closed by configuring field preprocessing and postprocessing, not by manual labor.
After a few weeks of operation, the share of manual corrections drops severalfold, and this is visible in the operations log, not in feelings.
Processing modes: cloud, on-device, hybrid
The choice of mode is determined not only by recognition quality.
The schemes behave differently on complex forms, treat personal data differently, and depend on the communication channel differently — and often the decision is made not for the entire app, but for individual document types.
In practice, the stream is almost never homogeneous: it is more efficient to route mass typical documents and rare complex forms through different pipelines. Below are three schemes and their real limits of applicability.
| Scheme | When it fits | Limitations |
|---|---|---|
| Cloud processing | Complex layout, handwritten and non-standard forms where maximum accuracy matters; moderate document flow | A stable channel is required: with a weak signal, response time grows and interruptions appear. The image leaves the device — explicit user consent and a storage policy on the service side are required |
| On-device processing | Typical documents with predictable layout, offline scenarios, data with increased privacy requirements | Accuracy on complex forms is lower. The model and its updates take up space in the build, and processing is noticeably slower on older devices |
| Hybrid scheme | Mixed flow: many simple documents plus rare complex ones; offline response needed without loss of quality | More difficult to develop and maintain: two recognition pipelines, switching rules, and unified logging of results must be tested together |
The hybrid most often wins: preliminary assessment and parsing of typical cases happen on the device, and only what fails the confidence threshold goes to the cloud.
Requirements for the communication channel are reduced — not every frame is transmitted, but only a few from the stream, so recognition remains operational even on unstable connectivity, and sensitive documents do not leave the device.
We fix the scheme with a “document type → processing pipeline” matrix and run it on a real stream of scans, not on demo samples.
Step-by-step implementation: from sample audit to handover
The work starts not from choosing a library, but from samples: without real frames, any accuracy estimate is a guess. That is why we start with a data profile and metrics and finish with acceptance on an independent sample. The order of stages does not change — each next one relies on the result of the previous one.
- Document sample audit. Real frames: different devices, lighting, angles, creases, handwritten inserts. They show which forms actually appear in the stream, not in the description.
- Reference markup. We manually mark up fields and character classes, and interpret disputed cases in writing. Then we count the share of correctly extracted fields, not abstract recognition accuracy.
- Choosing a processing scheme. On-device, server-side, or mixed: offline mode and data privacy pull toward the device, while heavy forms and frequent model updates pull toward the server.
- Configuration for form types. For each document — its own area of interest, preprocessing (alignment, contrast, noise), and postprocessing: database verification, checksums, parsing of ambiguous characters.
- Testing on complex cases. Regression set: crumpled receipts, shadows, rotations, mixed languages, handwritten edits. A failure is either closed by preprocessing or the scenario is sent to manual input.
- Pilot implementation. A limited group of users and collection of problematic frames from operation — this is how cases that were not in the original sample are caught.
- Quality acceptance and handover. Measurement on an independent set, fixed thresholds, and at low confidence — manual input instead of a silent error. We hand over configurations, markup, and regression sets.
After handover, the team retains a reproducible pipeline: new samples replenish the reference set, metrics are visible on the monitoring dashboard, and quality degradation is noticeable before users see it.
Configurations, schemes, and policies that remain with you
Recognition lives not in code but in configuration: parsing rules, confidence thresholds, and storage policy change more often than the module itself. That is why we hand over not only the build but also a set of artifacts that your team works with independently.
- A recognition profile for each document type: field composition, confidence threshold, and the rule for low confidence — accept, send for manual review, or request a re-capture.
- Parsing rules: how each field is extracted, which consistency checks are mandatory, and which errors block writing to the accounting system.
- Data flow scheme: capture, preprocessing, recognition, operator review, writing — with a note of where data leaves the device.
- Access and storage matrix: who sees images and results, how long they live, what is encrypted on the device, and what is erased immediately after parsing.
- Event logs: document type, profile version, field confidence, processing time, and failure code — they show whether accuracy dropped on a new form.
- Test sample set and acceptance policy: the share of fields parsed without operator correction and the threshold below which rolling out a new version is blocked.
- Retraining instructions and failure escalation procedure: how to add samples, check the profile, roll it back, and whom to contact in case of failure.
The artifacts are stored in your code repository and updated according to the policy: when the form format or storage requirements change, your team makes the edits without contacting us.
Case study: parsing invoices in a warehouse without manual input
What it was like before implementation
At the warehouse of distributor «Garant-Logistika», receiving looked like this: a storekeeper laid out paper invoices and entered items into a data collection terminal manually. A batch of 50–60 lines took 20–25 minutes, and with crumpled paper or low light — noticeably longer.
The share of lines that were later corrected manually stayed around one third: an error in the SKU surfaced during reconciliation.
Not only receiving time suffered. Disputed items accumulated until the end of the shift, and closing shifted by an hour to an hour and a half: instead of shipping, the crew sorted out discrepancies in quantity and units of measurement.
What was changed and what was achieved
The terminal camera became the main input channel, while manual entry was kept as a fallback. Before recognition, the frame undergoes preparation: desaturation, cropping along the document’s contrast border, and perspective alignment — you can capture at an angle.
Fields are found by the invoice template anchors, and the recognized SKU is checked against the item catalog.
Next are validation rules: quantity is an integer, the unit of measurement matches the product card, and the barcode checksum is verified.
Lines where something does not match are highlighted in the list, and the storekeeper corrects two or three items instead of the entire invoice. If there is no connection, images go into a local queue and are sent to the server when it appears.
The result based on quarterly measurements: the share of manual corrections — from 30% to 4% of lines, a batch of 60 lines is processed in 3–4 minutes instead of 25, disputed items per shift — from 25–30 to 4–6. Shift closing fits into 20 minutes because discrepancies are resolved at the moment of receiving.
How does on-device processing differ from server-side processing?
The difference starts with where the recognition model runs. With local processing, the camera frame goes into the model directly on the device, and the result appears in the interface without a network request.
With server-side processing, the frame goes to our pipeline, is recognized there, and the app receives a ready structured response.
Response latency is determined precisely by this. Locally, it is fractions of a second, and it almost does not depend on connection quality — recognition works in the subway, in a warehouse basement, and while roaming without data traffic.
The server-side scheme with a good channel gives comparable speed on a short document, but on a passport or a long invoice the time grows: frame-by-frame transmission and the processing queue are noticeable to the user.
On the other hand, if the connection breaks, the server-side scheme does not lose what was recognized — the request goes into a queue and is repeated, while the local one simply returns an error if the model could not parse the frame.
Privacy decides the choice more often than speed. If the document does not leave the device, the customer has no obligations for personal data protection on our side: no transfer, no storage, no logs with passport data.
Server-side processing requires a secure channel, encryption at rest, a retention period, and access control — this is manageable but separate work.
The schemes have different failure modes. The local model confidently handles printed text, receipts, barcodes, and typical forms, but stumbles on handwriting, rare fonts, complex table layouts, and low contrast.
The server-side one with a full-fledged engine handles handwritten input, hieroglyphs, and multi-page documents with tables — but breaks on a poor channel and with a large stream, when the processing queue grows faster than responses arrive.
In practice, we almost always arrive at a hybrid scheme: basic scenarios — contracts, receipts, employee documents — are recognized on the device, while handwriting and complex layouts go to the server if the user has consented and the network is available.
This way, choosing a scheme becomes not philosophical but engineering: each document type gets its own route, and the failure of one does not stop the other.
Limitations and common doubts: what we clarify before the start
Before the start, we record which scenarios the app must cover and where text recognition is known to be wrong. This removes the main risk — a mismatch of expectations: accuracy depends on capture quality, document structure, and data type, not only on the chosen technology.
Below are the doubts that most often come up during alignment and how we address them in the architecture.
How accurately are poor images recognized?
Before parsing, the frame undergoes preparation: document boundaries are determined, perspective and tilt are aligned, and shadows and glare are suppressed.
On clean images, accuracy for printed text is high; on crumpled forms and in low light, we set a confidence threshold and send questionable fields for manual review — the error does not move further through the process.
What should be done with handwritten entries?
Printed forms and machine-readable zones are recognized reliably; handwritten ones — noticeably worse and depend on handwriting.
Areas of interest are selected by template: critical fields are confirmed by an employee, and the rest are filled automatically with low-confidence highlighting.
Where is personal data from documents stored?
By default, images are processed on the device and do not leave it. If server-side processing is needed — we enable encryption during transmission and storage, a retention period limit, role-based access, and an access log, and we delete the original images after parsing.
How can a new form type be added without releasing a new version?
The template describes areas of interest and parsing rules, so a new form is often added by configuration. If the structure differs significantly, a labeled sample and model configuration are needed — we estimate its scope in advance, before work begins.
Let’s discuss the task: we will come back with an estimate and work plan
To make the project estimate substantive rather than “plus or minus an order of magnitude,” we need input from you: what documents are captured with, in what lighting, what form types and languages occur.
Text recognition is a task where accuracy is determined not by the model but by the quality of input frames and markup, so analyzing real samples gives more than any description of requirements.
What is needed from the customer for an estimate:
- 20–30 real capture samples — not scanned reference images, but what a phone camera captures in the field: with glare, tilt, shadow;
- a list of document types and languages, including handwritten fields and mixed layouts;
- usage scenario: who captures, how many documents in a row, whether there is a network connection;
- requirements for offline operation and for storing sensitive data on the device;
- where the result goes — to the interface, server, or accounting system, and in what form;
- constraints on timelines, platforms, and the app’s current codebase.
In response, we return not correspondence but a working package: the scope of work by recognition stages, the processing scheme — from frame to structured field, and a work plan with the start order and checkpoints.
Separately, we fix acceptance metrics: the share of correctly recognized fields on your set, processing time per page, and behavior at low frame quality. Describe the task — we will come back with an estimate and a pilot proposal.