On-Device ML: Training and Inference on Device Without Cloud

Why on-device ML does not reach deployment

A model leaves the lab with good metrics, and that is where the project usually stalls. On-device inference was tested on a single reference unit, where memory and compute resources are more than sufficient.

The real fleet consists of dozens of device models of different classes: what confidently passes on the reference device either does not start on half the fleet or loses quality.

The first bottleneck is memory constraints. Weights, intermediate tensors, and the application interface itself share one budget, so local model inference displaces neighboring processes or gets displaced itself.

Weight compression and reduced bit depth help, but each such decision costs accuracy, and it must be confirmed again on real data, not on a lab sample.

The second is response latency. On a device, the model does not run alone: it competes for compute resources and shares the budget with the main application, and under load the response time grows unpredictably.

The user sees “it lags,” the team sees an unmet response-time requirement, even though on the test bench everything fit with margin.

The third is fleet heterogeneity combined with offline operation.

The same build behaves differently on different device models, and updating the model where connectivity is absent for weeks is impossible: neither telemetry about degradation nor rollback after an unsuccessful weight replacement arrives. Support sprawls, and autonomy turns into a blind spot.

None of these problems is solved by a single technique: you need a resource budget for each device class, quality measurements under real load, and an update scheme with rollback. Below we break down how this is addressed in practice.

What local on-device model inference provides

Local inference means the model weights reside inside the application, and computation runs on the device itself — neural processing unit, graphics core, or central processing unit.

The model is compiled once for the target platform and quantized: artifact size is reduced several times, while accuracy on typical tasks remains acceptable.

Response stops depending on the communication channel. A small model responds in tens of milliseconds, and this time is the same with good and bad signal — the share of failures due to connectivity goes to zero, and the interface behaves predictably.

Offline operation becomes not a separate mode but a normal state: a warehouse, a trip out of town, a basement without coverage.

Server offload is the second half of the effect. Recognition, classification, and feature extraction are computed on the client, while the server side handles synchronization, delivery of model updates, and aggregation of anonymized statistics. Its capacity grows noticeably slower than the number of users.

Data privacy on the device is a consequence of architecture, not a promise: frames, audio, and text remain in the application memory and do not go over the network. Less transmission means a smaller leak surface and fewer obligations to store other people’s personal data.

Different accelerators are covered by TensorFlow Lite: one inference interface across multiple platforms, so changing the model or device does not require rewriting the inference layer.

Options for running an ML model: comparison of approaches

Running a model on a device is not one scenario but three different modes with different support costs. The choice between them is determined by where the data resides, how often the weights change, and whether the device has memory and power margin.

In practice, modes are combined: a compressed model handles baseline inference, while rare heavy cases go to the hybrid scheme.

A common intermediate representation such as ONNX allows transferring the same model between execution environments without retraining it for each platform anew.

The selection criterion is calculated by two numbers: fleet size and weight update frequency.

While the model is stable, the ready-made option is sufficient; as soon as regular fine-tuning or strict memory constraints appear, adaptation and the hybrid scheme pay off through fewer failures on devices.

Implementation stages: from scenario audit to handover

The order of work here is not bureaucracy but a way not to redo the model twice. On-device ML is limited by memory, heat, and battery life, so a solution built for an average device falls apart on a live fleet.

That is why we go from scenarios to hardware: each next stage relies on the measurements of the previous one.

1. Audit of the device fleet and scenarios. We collect models, OS versions, available memory, and places where the user is not ready to wait. The output is a list of scenarios the device can actually handle.
2. Inference scheme. We decide what is computed on the device itself and what goes to the server: a heavy model once a day or a compact one for every action. Separately, we record which data never leaves the device.
3. Model build and optimization. Weight compression, task-specific fine-tuning, caching of intermediate results. The goal is to fit into memory with margin, not tightly.
4. Testing on the real fleet. Warm-up, long session, heating, discharge, behavior under memory shortage. We look at the share of failed inferences and latency tails, not the average.
5. Solution handover. Code, inference scheme, model update procedure on devices, monitoring metrics.
6. Training the customer’s team. We walk engineers through how to retrain the model, how to roll back an update, and what to look at in the logs — after that the solution lives without us.

Skipping any stage comes back as rollbacks on the live fleet.

What remains with the team: configurations, schemes, procedures

An on-device solution works without the cloud, so all logic — model build, data access, update order — is handed over to the team as artifacts, not verbal agreements. Below is the full set that remains with you after acceptance.

• Model build configurations — files from which the build is reproduced from scratch: quantization parameters for target device classes, composition of exported layers, weight checksums.
• Integration scheme — where the model is called in the application: entry points, background operation, memory and power constraints, behavior under resource shortage.
• Data access policies — what remains on the device and never leaves it, what and in what form goes to telemetry, anonymization rules, local storage encryption.
• Model update procedure — how a new version reaches devices: rollout to a small share of the audience, stop criteria, rollback to the previous build without reinstalling the application.
• Solution documentation — architecture diagram, dependency composition, memory and thermal constraints, list of failure scenarios and what the user sees.
• Set of acceptance checks — a run on the real device fleet: response quality, response time, degradation under discharge and weak signal.

The set is designed for independent evolution: replacing the model, adding a scenario, or changing the storage policy can be done by your team without returning to us for every change.

Case: warehouse recognition without cloud connectivity

The warehouse environment consists of hangars, rack areas, and cold rooms where stable connectivity simply does not exist.

Recognition ran on the server side: a frame went over the network, the response came back with a delay, and during an outage the operator was left without a prompt and worked blind until the end of the shift. Inventory discrepancies accumulated by shipment — they were found during stocktaking.

We moved the model to the terminals themselves: inference runs locally via TensorFlow Lite, and the frame does not leave the device.

The cloud was left only for directory synchronization and accumulation of anonymized statistics — nightly packet exchange instead of a constant stream.

The network stopped being on the critical path: scanning, recognition, and prompts work without it, and accumulated records go to the server as soon as the channel appears.

What changed in the metrics

• Response: instead of seconds of waiting and queues at peak hours — a local response independent of network load.
• Availability: 100% of work scenarios execute without connectivity, previously only part, the rest waited for the channel.
• The failure class “server did not respond” disappeared from the operational summary; physical ones remained — dirty optics, consumable wear.
• Recognition accuracy after fine-tuning on real warehouse images: from ~92% to ~97% on the reference set.
• Inventory discrepancies by shift close decreased several times — reconciliation happens on site, not after the fact.

Resilience to connectivity outages

We separately tested recovery: the terminal survives network loss, restart, and discharge without losing accumulated operations — the queue is restored from the local journal. We did not have to rebuild the warehouse for coverage: the compute node is now the device itself.

How does local inference differ from cloud-based fine-tuning?

The boundary is not where the result is computed but where the model weights change. Local inference is a fixed model on the device: response in fractions of a second, offline operation, but it does not accumulate operational experience.

Server-side fine-tuning is the opposite case: weights are rebuilt from aggregated data, but every update must be delivered to the device fleet and compatibility checked.

The key question is where to store training data. Personal data (voice, images, messages) is better not to export from the device: then fine-tuning is federated, and only weight updates go out, not raw records.

If the data is anonymized, the server side gives more — the shared experience of all devices at once.

In practice, the hybrid scenario often wins: the base model on the device maintains fast response and offline operation, while the server fine-tunes personalization and distributes updates on schedule with rollback to the previous version.

We fix the choice of inference scheme before work starts: client code volume and which data leaves the device depend on it.

Technical concerns before starting work

Questions at this stage are usually about engineering, not the idea itself. Below is how typical concerns are addressed: fleet resources, model update without reinstall, inference quality variance, fine-tuning on small data, and embedding into an existing application.

Can the model be updated without a new application version?

We move the weights into a mutable artifact rather than embedding them in the build: the module is embedded into the existing application and checks the version manifest at launch.

The update is pulled in the background, switching occurs after checksum verification, and the previous artifact remains the rollback point.

Why do answers differ across devices?

Different accelerators and compression formats produce accuracy variance. We run a unified set of test scenes across the device matrix and look not only at answer match but also at stability on repeats; a branch that does not fit the tolerance is removed from the support matrix.

There is little data for fine-tuning — is it worth it?

Yes, if you do not train the whole model: freeze the main part, and fine-tune a small adapter on top of the base model on a modest labeled set. When there is no data at all, we start with collection — labeling disputed cases provides material for the next iteration.

Let’s discuss the task and come back with an estimate and plan

To come back with a timeline estimate and work scheme, not general words, we need your input on several points. Below is what to describe from the customer side: this is enough for us to assemble a plan and immediately determine how we will verify the result.

• Task and data: what signal is at the input (image, sound, text, sensor readings), sample size, whether there is a requirement not to take data outside the device.
• Target fleet: device classes and models, share of old versions, operating mode — single requests or a stream, memory and power margin.
• Quality: the metric by which we consider the solution working, acceptable error rate, and scenarios where an error is critical.
• Offline operation: whether a fully offline mode is needed, how the model updates on the device, and how it rolls back to the previous version.
• Integration: what on-device ML interfaces with — camera, microphone, application, onboard system, what accesses and permissions already exist.
• Acceptance criteria: a set of scenarios for the run on the agreed fleet and what we consider a completed stage.
• Pilot on a limited fleet: how many devices you allocate, over what period we collect measurements, and by what signs we decide to scale.

Then we come back with a work plan: data preparation and labeling, model selection for the constraints of a specific device, testing on the real fleet, update scheme, and result measurement by the agreed criteria.

Describe the task in free form — we will come back with an estimate and plan.