Implementing Pose Estimation in a Mobile App

Pose recognition in a mobile app: where accuracy is lost

Pose recognition in a mobile app is a pipeline of three links: camera frame, model, skeleton on screen. Accuracy is lost not in the model itself but at the joints — between frame and computation, between adjacent frames, between product expectations and the time budget on device. The metric on the test set holds, while in the user's hand the skeleton jitters and drifts.

Pose estimation errors are most often visible as jitter of keypoints. The model processes each frame independently, so detector noise on wrists, feet, and in overlap zones is not suppressed but transferred into coordinates.

Without temporal stabilization — tracking a point between frames, smoothing, prediction on a dropped frame — the joint angle is computed with variance, and rep counting starts to lie.

The second failure is a mismatch between video stream and model in time. The frame queue grows, computation uses an outdated frame, while the overlay is drawn on top of the current one: the skeleton arrives with delay and “floats” past the body.

For rep counting, these are lost movement phases. It is treated by discarding outdated frames by timestamp and separating the camera stream and computation stream.

The third is the budget of a weak device. The hardware cannot handle the stream, heats up, and throttles, after which the app either skips frames or changes the model class on the fly.

We calculate in advance how much time remains per frame and set predictable degradation: less frequent computation, interpolation between poses, honest interface behavior. We explain how this is handled in practice.

Measurable effect: what changes in the product after implementation

The first effect is visible to the eye — response latency. While skeleton overlay ran on the full frame, a queue of several frames accumulated between human movement and interface reaction: the user lagged behind the prompt, and the exercise had to be started over.

After moving the main branch to a lightweight keypoint model, response latency compressed to one or two frames — the interface reacts almost at the moment of movement, even when the person speeds up.

Resilience to frame loss is the metric on which a prototype usually falls apart. Previously one blurred or occluded frame reset the history, and rep counting drifted.

Now the pose is reconstructed over a temporal window: a dropout does not reset the trajectory, and the share of lost frames on fast movements falls from units of percent to tenths of a percent.

Measurements in MLPerf Mobile and our runs on live people converge: lightweight branches hold accuracy where heavy ones no longer keep up with movement.

Coverage of weak devices expands not “by eye” but by compute budget.

The freed headroom goes to skeleton smoothing and checking anatomical plausibility — pose recognition in a mobile app works on devices we previously excluded from support, and does not require a separate build for each model.

The main thing is reducing the class of failures: the mass category of “it doesn’t recognize me” requests goes away.

Frames with an incomplete figure, side lighting, or a person who has partially entered the frame no longer produce an error: the skeleton is completed from visible points with a confidence estimate, and a questionable pose is flagged gently — with a suggestion to adjust position rather than a failure message.

Model operating modes: on-device, cloud, hybrid

The three circuits differ not in “model quality” but in where skeleton keypoints are computed and how network, battery, and privacy behave.

The mode is fixed before development starts: app architecture, device requirements, and the list of scenarios worth promising at all depend on it.

Hybrid is not a “middle” option but a switchable scheme. We establish a unified set of classes and shared post-processing so that points do not “jump” when switching from cloud to device. The switch itself is governed by conditions: channel availability, charge, exercise type.

We validate limitations on a pilot set of real devices: the share of frames processed without delay, behavior on a weak channel, resilience to a partially occluded body.

Based on these measurements, we choose the circuit and fix it in technical documentation together with degradation behavior — what the user sees if inference is unavailable.

Implementation stages: from data audit to circuit handover

The order of stages here matters more than pace: if you start with the model rather than the data, the work returns to relabeling, and acceptance turns into an argument about numbers. We run the project so that each next step relies on an already agreed result of the previous one.

1. Data audit. We look at how users actually record: angles, lighting, occlusions, session length. The output is a list of scenarios the solution must handle and an estimate of labeling volume.

2. Labeling preparation. We label skeleton keypoints and fix rules for controversial frames — occluded limbs, a person leaving the frame boundary. We cross-check labelers against each other; otherwise the model learns from contradictions.

3. Circuit design. We decide what is computed on device and what on server, how the app behaves without network, and how to update the model without releasing a new version.

4. Model tuning. We fine-tune on our own corpus, separately test rare poses and extreme angles, and ensure accuracy does not drop on weak devices.

5. Optimization. We measure heating, battery consumption, and response latency on real devices: accuracy that makes the phone heat up does not go into the product.

6. Acceptance check. We run the agreed set of scenarios and metrics, check poor lighting and network interruption. A deviation from the threshold means returning to the appropriate stage, not “tweaking by eye.”

7. Circuit handover. We deliver integration code, circuit diagram, labeling and training scripts, model update documentation, and a diagnostics checklist, and train the customer's team.

Delivery composition: model configurations, flow diagrams, regulations

We hand over not a demo prototype but a working delivery that the support team can live with without us. Each artifact is tied to its own scenario: training on new data, rolling out an update, rollback, incident analysis on the customer's device. Below is what physically remains in your repository and knowledge base after handover.

• Model configurations: weights, confidence thresholds by keypoint, input frame resolution settings. Each parameter has a comment on what it changes and which metric it affects: false positive rate, resilience to a partially occluded body, performance in low light.
• Data flow diagrams: from frame capture to output skeleton. It is marked where the buffer is, where frames are discarded when there is not enough time for processing, what happens when the capture rate drops, and how the queue behaves under load.
• Access policies: which app modules access the model, what goes to the server side, and what is computed on device. Separately — rules for handling the user's video stream: where it is stored, how it is masked, when it is deleted.
• Model update regulation: the full cycle — labeling new recordings, fine-tuning, validation on the reference set, gradual rollout to a share of users. With rollback criteria: if the target metric drops, the previous version is restored with one command.
• Reference regression set: recordings with pre-labeled results, on which every new build is run. It allows catching degradation before release, rather than from support complaints.
• Profiles by device class: where the model processes frames in real time, where it switches to a lightweight variant, and where the logic degrades to analyzing individual frames instead of a stream.
• Support documentation: module map, extension points for new movement types, list of known limitations, and instructions for the on-call engineer — what to check first when metrics deviate from expectations.

Implementation case: metrics before and after under real load

Implementation case — a trainer app for home workouts: the phone camera tracks pose, and the system prompts whether the exercise is performed correctly.

At the start, the share of correct poses on the test set held around 68%, response time to a position change was about half a second. On fast movements the skeleton “jittered,” a squat was counted as a failure, and users abandoned the workout halfway.

What was changed in the model and processing

We rebuilt the labeling for real shooting conditions — different lighting, clothing, incomplete frame — and retrained the model with quantization for the mobile accelerator, so that inference is computed on the device itself, without going to the server.

We added keypoint smoothing across a sequence of frames and separate confidence thresholds for static and dynamic exercises: the skeleton stopped “jerking,” and the system learned to distinguish a pause between reps from a completed movement.

Metrics before and after

The share of correct poses on the same set grew from 68% to 94%, response time shortened to tenths of a second.

Measurements were compared against reference labels from the open COCO set, so the numbers are comparable between releases and do not depend on who exactly ran the test.

A side effect — the share of workouts completed to the end rose from 41% to 73%.

We do not count “by eye”: a fixed set, a run on three device classes, a report with error distribution by exercise type, and a regression test on every release — on the next update, metrics will not drift down unnoticed.

Do you need your own labeling if a ready-made model already works?

A ready-made model like MediaPipe is trained on mass data: a person at full height, camera at chest level, ordinary clothing, one person in frame.

As long as your scenario fits this distribution, fine-tuning adds almost nothing — keypoints are stable, and weeks of labeling are not needed.

Fine-tuning closes the bias but does not change the nature of the task.

If your movements are a variation of the general task (same person, same projection, different clothing or angle), it is enough to fine-tune the model on several hundred labeled frames and monitor quality specifically on your movements, not on a general validation.

Your own labeling becomes mandatory when the object of observation or the metric itself changes.

Inventory in hands, hand position during a grip, tempo and amplitude of movement — a ready-made model does not distinguish such differences: its target variable is “where is the joint,” while you need “is the movement performed correctly.” You have to label for your own task.

The cost of quality support is not calculated one-time. Pose labeling is tens of points per frame, consistency between labelers, a share of rechecks, a set of regression checks for already closed cases, and repeated fine-tuning on new data. Without this pipeline, quality on your movements drops with every scene change.

The practical rule is simple: first run the ready-made model on your recording and divide errors into systematic and random. Systematic ones are treated by fine-tuning; random ones and “wrong” movements — only by your own labeled set.

On a short pilot this is visible within days, and the decision relies on a metric, not on expectation.

Technical doubts: accuracy, privacy, failures

Doubts before starting are almost always the same: will the model handle poor lighting, will video leak, what will happen to the battery during a long session, and who is responsible for quality after launch. The answers determine what we compute on device and what goes to the server.

Does video go to the server?

Frames are processed on device and do not go outside — only point coordinates or aggregated metrics are transmitted. Server-side processing is enabled selectively, with user consent and channel encryption.

How long does the battery last?

Computation does not run on every frame: the stream is thinned, a reduced frame is fed to the model, and the camera turns off between sets. Heating and consumption are measured on real devices of the target lineup.

Who is responsible for quality after launch?

Accuracy does not drop by itself but from changing conditions — a new location, different clothing, a system update. Therefore the delivery includes a set of control recordings and a metric by which a drop is visible before user complaints.

Let's discuss the task: what data and access are needed to start

To come back with an estimate and implementation plan, a minimum of input data is enough for us — we will clarify the rest along the way.

The more precisely the scenario is described, the fewer assumptions in the estimate and the earlier bottlenecks are visible: for example, that some poses are indistinguishable in low light or that movement is too fast for real-time processing.

• Scenario: which movements and poses we recognize, how many people are in frame at the same time, what happens on error — prompt, pause, repeat exercise.
• Target platforms and minimum OS versions, list of device models on which the app must work stably.
• Video source: real-time camera, gallery recording, or external stream. If computation runs on device (for example, on MediaPipe), video does not leave the phone — this removes some privacy concerns.
• References: similar apps, examples of videos with movements, and an analysis of what you like in them and what does not fit your scenario.
• Access: repository, app store accounts, test builds, and a device fleet for testing on real hardware.
• Limitations: requirements for video storage, offline operation, battery consumption, and case heating during a long session.
• Deadlines and phasing: is there a hard date, is a pilot on a limited group of users needed before release.

Send what you already have — some items may remain open; we will assemble the estimate and implementation plan from the rest. Describe the task in free form: we will return with a breakdown by stages, timelines, and a list of what we will check in the first prototype.