ML Kit in an app: where performance breaks
ML Kit provides ready-made recognition modules — text, faces, barcodes, objects — and computes everything on device: the frame doesn't go to a server, and the result arrives without network latency.
In a demo with a single image this looks flawless, so ML Kit integration is often evaluated by first launch rather than behavior in a real scenario.
The first failure is calling recognition directly from the main thread. While the model processes a frame, the UI doesn't redraw: animation stutters, taps are lost, camera preview jerks.
The user describes this as "recognition lags," although the module itself works correctly — the execution thread is to blame.
The second is reinitializing the detector on every screen entry or app launch. Loading and warming up the model take noticeable time, memory for buffers is allocated anew, and the first frame after opening the camera waits for initialization. One long-lived instance with lazy creation is cheaper than a new object per screen.
The third is frames being fed into the module without a queue and at full resolution. The camera emits them faster than the module can process: the buffer grows, result lag from reality is measured not in frames anymore, the device heats up and throttles.
Hence the working set of solutions — a separate thread, dropping stale frames, and lowering resolution for the specific recognition task.
Mobile app performance here is determined not by the model but by the architecture around it: where the detector instance lives, which thread the result is computed on, and how many frames actually reach processing.
What local data processing on device gives
Local data processing on device means that recognition of text, objects, or faces runs on the phone itself, without sending frames to a server.
We build such scenarios on ML Kit: the toolkit works on top of the platform's system modules, so the model doesn't need to be bundled into the build and updated manually with a release. Below are two effects the customer notices first.
Interface response time
In the network variant, each frame waits for a server response: the camera already shows the image, but the recognition box appears with a noticeable delay.
With on-device processing, the result returns in the same tick as the camera stream — the UI responds without pause, and the hint doesn't lag behind hand movement.
Offline recognition also removes a whole class of failures: "service unavailable," long wait for a response, drop on a weak channel. The scenario works equally in the subway, in a warehouse, and in roaming, because it no longer depends on coverage.
Battery and traffic consumption
The price of local processing is load on the CPU and storage, and it shows in the app's battery drain if nothing is tuned.
We limit it deliberately: we reduce the frame analysis rate, separate heavy steps across threads, and unload the model when the screen is inactive or the camera goes to the background.
Then the app stays within the normal camera usage profile, rather than draining the battery in an hour of active recognition.
Traffic drops multiple times: only results go out, not a stream of frames, so on a weak channel the scenario continues to work, whereas the server variant simply stalls.
ML Kit on device, on server, and hybrid: what to choose
The choice of inference architecture is a tradeoff between privacy, response latency, and recognition quality. Built-in ML Kit modules on device cover typical tasks, while a server is needed where the model trains on your data and updates more often than the app release.
| Variant | When suitable | Limitations |
|---|---|---|
| Fully local processing | Offline scenarios, data doesn't leave the device, instant response without network | Typical set of classes, quality drops on weak devices, app size grows |
| Server processing | Custom models, frequent retraining, large class volume and rare events | Network required, response depends on channel, data leaves the device — consent and transfer protection required |
| Hybrid scheme with local prefilter | Camera stream where a small share of frames is of interest | Harder debugging: two validation loops and two versions of logic |
When a ready-made module is enough
Text, barcode, face, and object recognition are done "out of the box" by ready-made modules: classes are fixed, accuracy requirements are typical.
Before choosing, we run them on your real examples — if the share of correct answers stays at the required level, a custom model isn't needed.
When a custom model is needed
A custom model is needed when classes are specific — defects, industry documents, rare gestures — or data cannot leave the device. Then the model is compressed for target devices, and a server loop is connected for complex cases and fine-tuning on accumulated examples.
Integration stages: from scenario audit to handover
A mobile app audit starts not with code but with scenarios: what tasks recognition solves, where it's appropriate on screen, and what the user sees if the model errs. The output is a data flow map, quality requirements, and a list of devices on which the app must run stably.
- Requirements gathering. We fix scenarios, input formats, acceptable response latency, and UI behavior when a result isn't received.
- Data flow diagram. We determine what's processed on device, what goes to the server, and how source data is stored and deleted.
- Recognition module assembly. We connect ML Kit as an isolated layer — the app isn't tied to a specific model, and it can be replaced without rewriting screens.
- Measurements on reference devices. We measure response time, memory consumption, and heating on a typical fleet, not on one flagship device.
Testing on device fleet
- Testing on devices of different classes: weak and old models, different OS versions, shooting in poor light, rotations, and blur. Here failures invisible under ideal conditions surface — accuracy drop, UI freeze, accelerated battery drain.
Acceptance and handover
- Demonstration on agreed scenarios and project handover to the customer: source code, model update policies, metrics collection instructions, and an acceptance checklist. The result is code, test sets, and a clear process for releasing new versions without our involvement.
Implementation artifacts: diagrams, policies, regulations
App handover isn't a code archive but a kit your team or another contractor uses to release versions and update models without us. What physically remains with the customer:
- Data flow diagram: where the frame or signal comes from, what's computed on device, what goes to the server.
- Build configurations and environment profiles: keys, variables, differences between staging and production environment.
- Model access rules: who publishes a new build, how it gets into the app, and how it's rolled back.
- App documentation: module map, ML Kit integration points, permissions, offline behavior.
- Source code handover: repository with history, release branches, rights to model weights, build from a clean environment.
- Release checklist: pre-publication checks and tests on weak devices.
- Maintenance documentation: log and metric analysis, typical incidents and actions for them.
Model update policy
The model lives separately from the code: data drifts, recognition quality degrades, while the app formally works.
The policy describes the full cycle — who initiates retraining, on which set the new version is validated, to what share of users it's rolled out first, and by which metric a decision is made to return to the previous one.
It also fixes the frequency of quality checks and the threshold at which an update is mandatory — so app maintenance doesn't turn into reacting to complaints.
Case: offline document recognition in a warehouse
At receiving and shipping, an operator with a terminal reconciles invoices, and the network deep in the hangar disappears exactly where work happens.
We moved document recognition onto the device itself — based on ML Kit, without sending images to an external service: camera, local model, ready fields in the accounting system.
What it was before
The document was recognized on a server: the terminal sent a photo, waited 4–6 seconds for a response, and in no-coverage zones the scan didn't go through at all.
The operator either stood by the partition with a signal-catching router or entered fields manually — hence quantity discrepancies and repeat receiving the next day.
The share of successful recognitions stayed around 70%, disputed documents weren't flagged in any way, and support received 25–30 tickets per week.
What it became after
The model works with the frame locally; the network is needed only to send already prepared fields. The share of successful recognitions grew to 97%, response dropped to fractions of a second, and receiving one vehicle was reduced from three minutes to forty seconds. Support tickets remained 3–4 per week — and those were about a lost terminal, not recognition.
We separately covered no-connection behavior: the document is processed and placed in a local queue, and fields go to the server when network appears.
Disputed fields are highlighted by recognition confidence — the operator sees what exactly to recheck, and the document doesn't enter accounting with an error. Document images do not leave the terminal.
How is a ready-made model different from a custom one?
A ready-made model from ML Kit is trained detectors for typical classes: faces, text, barcodes, objects, language. As long as the task fits this list, training isn't needed: integration reduces to a call, and recognition accuracy on typical data is predictable.
A custom model is needed where classes are specific: line defects, rare document types, industry terms.
The difference shows on customer data. A model trained on a general sample loses accuracy on atypical scenes, non-standard layouts, and rare wording.
A custom one pulls these cases, but requires labeling — several thousand examples per class plus a validation set that doesn't participate in training.
| Criterion | Ready-made model | Custom model |
|---|---|---|
| Accuracy on typical data | High without tuning | Not higher until own data exists |
| Accuracy on customer data | Drops on atypical scenes | Grows with each fine-tuning cycle |
| Time to first version | Days for integration | Weeks for labeling and training |
| Labeling requirements | Not needed | From several thousand examples per class |
| Data entry threshold | Zero | Accumulated archive required |
| Maintenance | We monitor external updates | Retraining when data drifts |
The choice isn't binary. We start with a ready-made model to validate the scenario with live users, and in parallel collect a data corpus.
When accuracy stops being enough, we fine-tune on customer data and compare metrics on the same set — the decision is made by numbers, not taste.
Technical doubts before integration
Before on-device recognition integration, the team has the same set of questions: how to update the model after build publication, what to do with a fleet of old devices, where user data goes, and whether the solution will handle non-standard documents. Briefly answer each — without marketing.
How to update the model without releasing a new build?
The model and app logic are separated: weights are in a separate downloadable module, and the app checks the version manifest and pulls the current file. The switch happens on the next launch, and rollback is by checksum to the previous version if accuracy drops on live data.
What about devices without hardware acceleration?
For them, CPU processing mode is enabled: input is reduced, frames are thinned, and heavy scenarios move to the background. We select thresholds by profiling on a weak fleet, not "by eye" — the report shows response time by device groups.
Where does user data go?
In the basic scenario, frames are processed on device and don't leave it. If server mode is needed for complex cases, we mask unnecessary parts, send only a fragment, and keep a journal without personal data — under information protection requirements.
What to do with non-standard documents?
Scans, faxes, handwritten forms, and multi-page files are recognized worse than typical ones. Before feeding into ML Kit, preprocessing runs: alignment, discoloration, noise suppression; for a specific template we add our own samples. This raises accuracy where the base variant errs.
Let's discuss the task and come back with an estimate and plan
So that the app project estimate relies on facts, not general words, at the input enough is what you already have: a scenario description, several examples of real data, and a device list.
This is enough to estimate which ML Kit capabilities cover the task on device and where server processing will be needed.
Next is a technical interview with the engineer who leads this direction: we break down the scenario, contentious points, and limitations, after which we come back with an implementation plan and timelines by stages.
- Usage scenario: what exactly should be recognized or detected in the frame and what happens with the result next — show it to the user, write it to a local DB, or send it to the server.
- Data examples: 10–20 real photos or video frames in the conditions where the app will work — different lighting, tilt, blur, glare on the screen.
- List of target devices and minimum supported OS version: this determines which models to pull onto the device and which to keep in the cloud.
- Current measurements, if a prototype is already assembled: frame processing time, memory consumption, build size increase, app behavior on weak devices.
- Privacy and offline mode requirements: should user data remain on the phone or is transfer to a server acceptable.
- Timeline and release window constraints: by what date the build is needed in stores, is there a tie to a marketing campaign or partner launch.
- What's already used in the project: a ready-made on-device model, a custom trained one, a third-party service — so as not to rewrite working parts unnecessarily.
Describe the task in free form — we'll come back with an app project estimate and a stage-by-stage implementation plan.