Image segmentation in a mobile app: where it breaks
Image segmentation in a mobile app almost never breaks “in theory” — it breaks under specific shooting conditions.
In low light, the sensor produces a noisy frame, the color and brightness threshold stops separating object from background, and the object mask on the frame tears along the contour: the user gets a cut-out silhouette with a missing sleeve, shadow, or part of the hairstyle.
A separate problem is semi-transparent and thin details: glass, animal fur, bicycle spokes. A model trained on “clean” studio examples simply does not distinguish such boundaries.
The second failure point is frame processing latency.
If the model processes a frame longer than the user is willing to wait, the segmentation effect loses its point: the person has already pointed the camera at another object, while the previous mask is still stuck on the screen.
Worse is when heavy computation runs on the same thread as preview rendering — the interface starts stuttering, buttons do not respond, and the app looks broken even though the code formally works.
The third failure manifests where it is least expected: in a field, in a basement, on an airplane.
Engineers often build a scheme where “the frame goes to the server, and a ready mask comes back,” and then the app’s operation without a network turns into a frozen screen with an endless spinner.
The user does not understand what happened, closes the app, and never returns — while in the logs it looks like a connection loss, not a product defect.
And the fourth layer is the quality of boundary annotation. It is by the mask edge that you can see how carefully the entire pipeline is assembled: from preparing training examples to smoothing the contour during rendering.
A ragged, jagged boundary, a “halo” around the object, and mask jitter from frame to frame irritate more than the segmentation error itself. Below we discuss how this is addressed: where to run the model, what to do when the network drops, and how to maintain a stable edge.
What segmentation on-device and in the cloud provides
The difference between computation on the phone and on the server is visible not in mask quality, but in how many frames make it to a response.
Five effects are measurable on real traffic: the share of frames without a network, interface response time, result consistency across different phone models, elimination of “no response from server” failures, and the volume of transmitted frames.
On-device processing: what goes offline
The model is compiled into the platform format — via Core ML, inference runs on the phone itself, without contacting the server. Everything that does not require heavy context goes offline: person, object, background, boundaries.
This is 80–90% of the frame stream: the app works without a network, and the interface responds in fractions of a second instead of seconds of waiting.
Consistent results across different phone models are provided by fixed preprocessing: one input size, one normalization, one mask threshold. Differences remain in speed, not geometry — this is verified by comparing the share of matching pixels across a matrix of devices.
Hybrid: where to compute and what to store
Edge-case frames go to the cloud: small objects, heavy occlusions, non-standard scenes — there the model is larger, and the result is more accurate.
This selection reduces the volume of transmitted frames severalfold, and the “no response from server” failure class disappears: the local mask is delivered immediately, and the refined one replaces it later.
It is enough to store masks and features, not the frames themselves — this removes privacy concerns and keeps the send queue predictable. Frames that did not go through over the network wait in a local queue and are sent again.
Computation options: on-device, on-server, hybrid
The computation mode determines where the model lives and how many resources it takes from the user.
Local frame processing runs directly on the phone and does not send the image to the network; server-side frame processing offloads the device but ties the result to the communication channel; a hybrid scheme splits the task between them. The choice affects response latency, privacy, and which device fleet the app can even run on.
| Option | When it fits | Limitations |
|---|---|---|
| Local frame processing | The mask is needed instantly, frames come as a stream, images are sensitive and must not leave the device | Higher device requirements: the model keeps memory occupied at all times, memory and thermal constraints cut off older fleets, and quality on complex scenes is lower than server-side |
| Server-side frame processing | Maximum accuracy is needed, the model is heavy and updated frequently, shooting is processed in batches rather than in real time | Latency depends on the network, transmitting every frame consumes traffic, and there is a separate storage perimeter and protection for personal images |
| Hybrid scheme | A balance is needed: a fast draft result on screen and precise refinement where it matters | Harder to maintain: two computation branches, mask format reconciliation, and control of model version desynchronization between client and server |
When on-device processing is enough
If the task is to separate an object from the background in real time and the scene is uniform, on-device computation wins: response does not depend on connection quality, and frames do not leave the phone.
We check device requirements against the target fleet in advance and select a lightweight model so that it fits memory constraints without dropping frame rate.
Hybrid: preprocessing on-device, refinement in the cloud
A fast pass runs on the phone — it produces a draft mask and filters out the unnecessary, while heavy refinement goes to the server on request.
This removes the main load from the device and reduces the volume of transmitted data, but adds the obligation to monitor model version compatibility on both sides.
How implementation proceeds: from data audit to handover
We build the implementation of segmentation into a mobile app so that each step relies on the verified result of the previous one.
First, we establish which object classes need to be segmented and under what conditions the user shoots — both annotation and model requirements depend on this.
- Frame and annotation audit. We check scenes, lighting, angles, and class balance, and separately look for discrepancies between masks and actual object boundaries: the model reproduces annotation errors rather than correcting them.
- Processing scheme. We decide where computation runs — on-device, on-server, or hybrid — how the frame is prepared before inference, and what the app does with the ready mask.
- Model assembly and training. We fine-tune a pretrained backbone on annotated frames, separately run rare classes and complex scenes so that the segmentation contour works not only on typical examples.
- Testing on real devices. We look not only at mask accuracy but also at memory consumption, response speed, and behavior in low light — some defects are visible only on a live device.
- Quality measurement. We calculate metrics on a holdout set and on recordings from devices, analyze frames where the mask tears, and use them to refine annotation and fine-tune.
- Handover to the customer’s team. We deliver the model, frame processing code, instructions for fine-tuning on new data, and the verification procedure before publishing a release.
What we clarify before work starts
Before starting, we clarify three points: where inference runs, how much frame data is available for fine-tuning, and what acceptance looks like — on which sample and by which metric we measure quality. The answers determine the scope of work and timeline.
What remains in the project: models, annotation, regulations
After segmentation is put into production, the customer receives not only the app but a set of artifacts that determine how the model behaves on new images. Below is exactly what we leave in the project and why the system loses controllability without each item.
- Model and its configuration — weights, architecture, input resolution, confidence threshold, and mask post-processing parameters, plus a build script so the result can be reproduced on your test bench.
- Annotated dataset — source frames and masks, rules for disputed areas: semi-transparent edges, occlusions, small objects; a revision log with author and date.
- Frame processing scheme — the path from camera to mask: resizing, normalization, on-device computation or server upload, and smoothing boundaries between frames.
- Retraining regulation — which metric and what drop in it triggers fine-tuning, how often we check on a control set, and who prepares new data.
- Team instructions — how to set up the environment, run quality checks, compare a version with the current one, and roll back if the metric drops.
Rights to data and model
Separately, we establish who owns the images, annotation, and trained weights, and which open models were used under which licenses.
If frames contain people or documents, we describe anonymization, retention periods, and the procedure for deletion on request — this way, project handover does not create legal risks.
Case study: object masks in an app for warehouse shooting
The client is a chain of distribution centers, SkladLine, and the task is inventory tracking using shots from the camera.
The project started at four sites: an operator pointed the camera at a pallet, the app segmented a mask for each box, and the result went to the inventory system.
The first version computed everything on the server — the frame was compressed, queued, and returned with a ready mask.
In areas with unstable signal, this resulted in stuck captures and shift rework: measurements before implementation yielded 74% correct masks, about 0.9 s per frame processing, and up to five server requests per shot.
What was changed in the processing scheme
The main work was moved to the device: a lightweight model computes the mask directly on the phone, and a frame goes to the server only when confidence is below the threshold — roughly every twentieth shot.
Frames are stored in local storage and sent automatically when the connection returns, so a network drop no longer interrupts shooting. The server retained recalculation of complex frames and accumulation of annotations for fine-tuning the model on data from a specific warehouse.
After the transition, the share of correct masks grew to 96%, frame processing time dropped to 0.2 s, and the number of server requests dropped to one per ten shots.
Peak load on the server side fell by almost three times: the same capacity now handles three times as many simultaneous captures, and complaints about freezing when the connection drops disappeared from support reports.
How does semantic segmentation differ from object detection with bounding boxes?
A detector answers the question “where is the object”: the output is a rectangle and a class. Semantic segmentation labels every pixel in the frame, so the result is a pixel-wise mask that follows the contour.
For a person’s silhouette, a bounding box will capture both the background and the arms, while the mask will follow the boundary precisely. This leads to different requirements for annotation, model size, and on-device computation.
Bounding boxes are sufficient when the result is counters, movement tracking, zones of interest, or rough analytics: annotation is collected several times faster, the model is lighter, and it runs more stably on the phone.
A mask is needed where the contour itself is the result: background replacement, virtual try-on, overlay graphics, object cutout.
A separate fork is similar objects. Semantic segmentation will merge two cars into one class; if you need both to count and to outline them, that is a third formulation, with a separate mask for each object.
The choice comes from the question “what exactly will we return to the app,” not from how pretty the picture is.
If the app already uses bounding-box detection, there is no need to rewrite it. We fine-tune the existing model: the detector remains, a mask prediction branch is added to it, and training runs on a small set with precise contour annotation.
The external contract in the app does not change — the interface receives the same result plus a mask, and while the mask is uncertain, we return the bounding box: the user does not see a failure.
We check mask quality with intersection metrics against ground truth — on public datasets with double annotation, such as COCO, and on our own frames from user cameras. Analysis of worst cases happens before enabling the mode on all devices.
Technical concerns: accuracy, devices, data
Segmentation breaks not on large objects but on thin structures. We fix the metric before starting: mask intersection share and correct pixel share for each class, with acceptance on a challenging set rather than studio frames.
Will accuracy be enough on complex frames?
Thin boundaries — hair, highlights, semi-transparent objects — require high-resolution input and contour post-processing. We fine-tune on shots from target devices and the same lighting conditions.
What to do with weaker phones?
A lightweight model branch: smaller input, weight compression, some operations on a simplified contour. The mask is computed at reduced resolution and refined at the edges — the interface remains responsive.
Who owns the data and the model?
Annotation, weights, and training configuration are transferred to the customer. If images must not leave their perimeter, training runs on their infrastructure, and we work with anonymized metrics.
What about rare classes?
We balance the sample and add rare-class objects to frames synthetically, and set the confidence threshold for them separately. The class is not shown in the interface until the metric on the holdout set is reached.
Quality drops after a device fleet update?
We maintain a regression set of reference frames and run it on every model build. A drop is visible before release: we roll back the weights, and investigate the discrepancy between device models using the report.
Let’s discuss the task: what to send for an estimate
To come back with an estimate and work plan, you do not need a volume of technical specifications — five inputs are enough. From them we understand the scope, the order of stages, and where the project may go off plan.
- Frame examples — 10–20 real shots or short videos from working devices. Not only successful ones matter: backlight, blurred background, similar objects nearby, shooting at an angle — these are exactly where the mask most often tears.
- Shooting scenarios — where the frame comes from: gallery, in-app camera, front-camera stream, offline operation. This determines where to compute the mask and how often.
- Device constraints — minimum supported OS version, share of weak models in the audience, acceptable frame processing time, and memory headroom.
- Privacy requirements — whether the frame is processed on-device or goes to the server, what is stored in logs, and whether anonymization of faces and license plates is needed.
- Object classes and accuracy — what exactly we segment, what error share is considered acceptable, and how the app behaves on an uncertain result.
- Timelines and milestones — by what date a working prototype is needed, what goes into the first priority, and what can be moved to the second.
After that, we will come back with an estimate and work plan: stages, order of checks, and points where you accept the result. If some data has not been collected yet, describe the task in words — we will help formulate the requirements on a call.