Edge AI development: where projects break on devices
Edge AI development almost always starts the same way: the model converges on the training set, but on the device it does not fit into available resources.
A neural network on a microcontroller is not a scaled-down copy of a server one: device memory limits, weight size, power and thermal budget are fixed in advance and do not expand as the project progresses.
The first failure is visible already on the test bench: the model's operation on the device fails on response time, the enclosure heats up, the controller throttles, and autonomous operation drops from weeks to hours.
The second failure comes with the data: vibration, sensor noise, calibration drift, and temperature degradation push the signal outside the training distribution, accuracy sags, and the model confidently answers where it should stay silent.
Compression for the sake of saving space finishes off what remains: quantization to integers breaks rare but critical classes, pruning removes exactly the feature that carried the useful signal, and window averaging smooths short events.
Edge AI implementation is a rebuild of the entire chain: problem formulation for the real scenario, feature selection for a cheap sensor, distillation into a compact network, fine-tuning on the target domain, and sleep mode with interrupt wake-up.
Below we break down how this is closed at each step and where such projects usually stop.
What on-device model inference provides
On-device model inference changes the very nature of response time. With cloud inference, latency is made up of the channel, the server queue, and the return path—on unstable connectivity it grows severalfold and unpredictably.
Local computation on a microcontroller removes these components: decision response time becomes deterministic and fits into the control loop rather than into waiting for the network.
Autonomous operation without a network is the second effect the customer notices immediately. The device continues to compute its scenario when the channel is unavailable: in a basement, in a field, on mobile machinery, in an isolated area.
The failure class "connection lost—function stopped" leaves the architecture instead of being masked by request retries and buffering.
Local data processing keeps the raw signal within the customer's circuit: only the result goes outside—an event, class, metric, not a stream of raw data.
On Arm cores, such inference works without the cloud, and the model is quantized to the memory budget of a specific microcontroller, so the solution does not require an external compute unit next to the sensor.
In sum, this is predictable response time, independence from connectivity, minimal transmitted data, and the removal of the channel-related failure class.
Traffic drops by orders of magnitude: instead of a continuous stream of measurements, rare events are sent, while the computational load remains on the device.
Implementation approaches: comparison of options
The choice of implementation option is determined by two things: how much the customer's data differs from public datasets and whether the scenario fits into the device resources.
For typical tasks—command recognition, defect detection—a ready-made model with fine-tuning on a labeled sample gives a result without training from scratch.
Comparison of Edge AI approaches starts with measurement: how many classes need to be distinguished, how quickly the situation changes, and whether it is acceptable to send data to a server. Based on these answers, the inference scheme is selected.
| Option | When it fits | Limitations |
|---|---|---|
| Ready-made model with fine-tuning | Typical scenario: classification, detection, command recognition. Need a fast start | Accuracy is limited by the original architecture; rare classes are recognized worse |
| Custom model for customer data | Unique objects, specific defects, non-standard shooting conditions. A labeled archive exists | Longer development cycle; labeling and its verification by a subject-matter expert are needed |
| Hybrid inference scheme | Heavy scenarios—large images, complex logic. The device is already networked | Dependence on the communication channel; part of the data leaves the circuit |
| Accelerator module | Autonomy and performance headroom for several scenarios at once are needed | Higher power consumption and enclosure requirements; platform choices are narrower |
In practice, the options do not exclude each other: we start with a ready-made model as a baseline, take metrics on real data, and use them to decide whether a custom architecture is needed or part of the computations should be moved to a server. The decision is made based on measurements, not assumptions.
Development and on-site implementation stages
Edge AI development is work within the constraints of the device itself: memory, power, response latency.
An error in a model that has gone into firmware cannot be fixed on a server, so the project follows a fixed sequence of steps—we run Edge AI development stages so that each ends with an artifact that can be reviewed and challenged.
- Device and data audit. We review the fleet of controllers, available memory, and power constraints; examine the sensor stream and accumulated recordings: volume, labeling, class imbalance. The output is an understanding of which scenarios are feasible on the current hardware and where a second processing circuit is needed.
- Solution scheme. We fix what is computed on the device and what goes outside, which input features we take, and by which metric we evaluate quality. The memory and response time budget is set in advance; otherwise training proceeds in isolation from the real hardware.
- Model training and compression. First we build a baseline model, then reduce it: pruning insignificant connections, lowering weight precision. At each step we compare quality against the original—the goal is to fit into the microcontroller memory with minimal accuracy loss.
- Hardware tuning. The model is embedded into firmware; we take actual latencies and consumption, check behavior under power sags, sensor noise, and peripheral failure.
- Bench and live data validation. We run recorded sessions and fresh recordings from the site, see which classes the model confuses. We select the trigger threshold by the cost of a miss and a false alarm, not by average accuracy.
- Solution handover. We hand over firmware, source code, scripts for reproducing training, and documentation; we train the customer's team—the next model version is retrained without us.
Delivery package: models, schematics, and regulations
The project ends not with a demonstration but with the handover of a circuit that the customer maintains on its own—without access to someone else's closed environment and without being tied to a single contractor. Here is what physically remains in your hands after acceptance.
- A trained and optimized model—weights in an open format, training and validation scenarios, a labeled dataset, and a quality map by class.
- Schematics and configurations—the electrical schematic of the unit, sensor placement and power, thermal mode, settings files for the lab bench and field devices.
- Data access policies—roles, where raw data is stored, what is anonymized directly on the device, retention periods, and export procedure.
- Model update regulation—how to build a candidate, what checks it passes before rollout, how to deliver it to devices, and how to roll back to the previous build.
- Handover of source materials—repository, build scenarios, and build parameters so firmware can be reproduced by the customer's team, not only by our engineers.
- Documentation set—model passport, applicability conditions and boundaries beyond which quality drops, unit operation manual.
- Integration scheme—where recognition results go, in what format, what happens when communication is lost, and how to restore the stream after a failure.
- Acceptance checklist—how to check the unit after sensor replacement or restart and which metrics to look at to distinguish degradation from a one-off disturbance.
Case: quality control on a production line
Before implementation, line control relied on sampling: an operator took several items per shift, and the rest went into the flow.
A deviation was detected at the end of the shift, when the spread had already extended across dozens of units, and when external connectivity sagged, the area switched to manual mode and lost pace.
We moved inference to the area itself: the model was trained and tested on a labeled defect archive, then quantized and assembled into an image for a microcontroller in a camera by the conveyor.
Frames do not leave the shop, and only a summary of deviations and updates under the regulation go to the server—with rollback to the previous image.
| Metric | Before | After |
|---|---|---|
| Control coverage | sampling, several items per shift | the entire flow in transit |
| Moment of deviation detection | at the end of the shift, after batch accumulation | on the item itself |
| Dependence on external channel | critical: outage means manual mode | none, inference is local on site |
| False alarms | frequent, generic thresholds | isolated, calibrated for the shop |
| Introduction of a new SKU | retraining from scratch | fine-tuning on accumulated frames |
The main effect is that the failure class "detected when the batch is already assembled" is gone.
Thresholds are calibrated for the lighting and speed of the specific shop, so false alarms are fewer, and the line no longer depends on external connectivity: when the channel goes down, control continues.
— Previously we learned about a problem from the quality control department, now from the line itself, — chief engineer at Svetlogorsk Machine-Building Plant.
When on-device inference loses to the server
Short answer: local inference yields to the server where the volume of context, the number of modalities, or the model update frequency grows. The device wins on a narrow task with constant conditions—event detection, recognition of a limited set of commands, signal filtering.
As soon as a long sequence, multiple data sources, or text generation appears, the memory and power budget runs out before accuracy reaches the required level.
We classify multimodal processing, long context, model ensembles, and everything where the model must be fine-tuned on a stream of new data directly on site as heavy scenarios.
A separate category is tasks where post-hoc decision analysis is needed: on a server it is easier to store input data, the model version, and the result, whereas on a device the log quickly hits the storage volume limit.
There is also an operational boundary. A fleet of devices updates more slowly than a service: a new model version is rolled out in waves, compatibility is monitored, and rollback is kept in case accuracy drops on some objects.
Debugging on a server is cheaper in time—the input data distribution is visible and a failure can be reproduced; on a device only the local log is available.
Therefore, at the start we count not "device or server" but the shares of processing. We look at four values: allowable response delay, channel stability at the site, what share of cases a lightweight model confidently covers, and how often it will need to be retrained.
If it filters out most of the stream, the remainder goes to the server—the channel load drops by an order of magnitude, while decision quality remains server-grade.
The boundaries of applicability for local inference are set not by the compute unit but by the stability of the problem formulation. The less often data and rules change, the more sense it makes to keep the model on the device.
A hybrid scheme almost always wins over extremes—we calculate it for a specific site and fix exactly what goes to the server.
Questions asked before start
Before an edge AI project starts, customers ask not about the choice of board but about what will happen after handover: whether the solution can learn on their data and who is responsible for it afterward. Below are answers to doubts that we remove before signing the contract, not during the work.
Can you fine-tune the model on our data?
Yes. The model learns on your sample, not on a public one. The fine-tuning cycle is built in from the start: your examples, a separate control set, and validation on rare classes—otherwise the first non-standard batch will tank accuracy.
Accuracy at acceptance is lower than expected—what next?
We analyze exactly where the model errs: rare classes, sensor noise, edge cases that were almost absent from the training set. Some discrepancies are closed by fine-tuning, others by tuning signal digitization before it reaches the model.
How do you update models on already installed devices?
The update arrives over the network in stages: first a small group of devices, and if metrics have not dropped—the rest. The previous version remains on the device as a rollback point. The build is in the portable ONNX format, so there is no need to travel to the site for reflashing.
Who labels the dataset?
We label it; your specialists review only disputed cases—usually a fraction of a percent of the sample. Pre-labeling is done by the model, so manual work remains several times less than when labeling from scratch.
What is included in support after handover?
We hand over a support regulation: field accuracy monitoring, fine-tuning on new data, fleet updates, and incident analysis. The solution remains operational even after device replacement or changes in line conditions.
Let's discuss the task and come back with a work plan
An accurate estimate for a microcontroller project can only be obtained from inputs on a specific site: generic approximations here give a spread of several times. To discuss the task substantively and avoid guessing, it is enough to send the following.
- Description of the site and scenario: what we measure, under what conditions, what counts as a trigger and what as a false one.
- Device fleet: controllers, sensors, power scheme, and available memory for the model and buffers.
- Data examples: raw sensor recordings over several days, including noise, anomalies, and power outages.
- Response and autonomy requirements: allowable reaction time, sleep mode, operating time from one battery.
- Constraints: operation without a network, data privacy on the device, requirements for enclosure, certification, and integration with the current system.
- Acceptance criteria: share of correct triggers, allowable misses and false alarms on a labeled sample.
- Timeline and staging: by what date a working prototype on a real device is needed.
In response, we run a candidate model on the target controller, check whether it fits into memory and the time budget, and estimate the labeling volume and labor costs for on-site fine-tuning.
We come back with a project estimate, a draft Edge AI technical specification, and a staged implementation plan: from dataset collection to serial firmware with over-the-air updates.
Describe the task in free form—we will come back with an estimate, a work plan, and the questions that need to be closed before start.
