Face tracking in AR: accuracy, animation and performance

Face tracking in AR: where facial expression accuracy is lost

Face tracking in AR is a contour of several dozen keypoints and a set of facial expression coefficients that are recalculated on every camera frame. The user sees only the result: the avatar blinks, mirrors a smile and head tilt.

As soon as point estimates lose stability, the product impression breaks — the mask “floats,” virtual makeup slides, and the character looks lifeless.

The first common failure is keypoint jitter. On a static face, the contour begins to ripple finely: smoothing cannot keep up with frame noise, and the eyelids or mouth corners twitch at a frequency the eye reads as a defect.

This is most noticeable in subtle facial channels — squinting, a raised eyebrow, slight smile asymmetry — where the cost of being off by a couple of points is immediately visible.

The second is facial expression sticking when the head turns.

In profile, some points are lost, and if the filter does not distinguish “face turned” from “face disappeared,” the facial expression weights remain from the last reliable frame: the user turns away and comes back, while the avatar still holds a half-smile.

This kind of facial animation failure breaks both dialogue with the character and video recording, which is often the reason the app is opened.

The third is contour breakdown in low light and on inexpensive devices: contrast drops, the number of confident points decreases, and jumps appear between them.

Here controlled degradation works — first we give up fine facial expression detail, then switch to coarse articulation, but do not lose tracking entirely.

This costs more than it seems: an unfinished try-on, closing the mask, a review saying “the face twitches” — direct losses in conversion and trust in the product.

Below we break down what stability consists of: calibration, filtering, pose prediction, and the time budget for frame processing.

What calibrated facial animation delivers: effect in measurable metrics

Face tracking stability is not a single algorithm but a combination: keypoint detection, temporal filtering of coordinates, and pose prediction between frames.

The raw point stream jitters, loses eyebrows and mouth corners during a fast head turn, and the tracker silently outputs the last valid state.

That is why in projects we look not at “average accuracy” of face tracking, but at the share of frames with correct pose and the length of continuous segments without dropouts.

What we fix as the result

• Share of recognized frames in operating conditions — from 99%, including head turns and partial occlusion of the face by a hand or hair.
• Stable facial animation during turning and tilting: articulation and eyebrows do not “snap” to neutral, but degrade smoothly according to a clear scenario.
• Operation in low light: filtering over frame history keeps tracking where the face can no longer be detected from a single frame.
• Predictable behavior across device classes: instead of a break, a reduction in update rate and gradual narrowing of the set of facial channels.
• Less manual animation cleanup: the share of frames an animator corrects by hand drops from a third to a few percent.

We treat the behavior of platform trackers such as ARKit as a baseline source, not as a turnkey solution: stability comes from a processing layer on top of it — smoothing with adaptive inertia, recovery of lost points from neighboring frames, and an explicit “face lost” state instead of false coordinates.

For the client, the outcome is that facial animation comes from real video without frame-by-frame cleanup, and the result is equally explainable on all supported devices.

Face tracking modes: comparison of approaches and limitations

The choice of mode is determined not by the device class, but by where latency is acceptable. On-device processing provides a predictable response without a network; remote frame processing opens access to heavy models but ties facial animation to channel quality.

On a weak device fleet, the difference shows up in the very first scenario — a long session with gestures and conversation.

A hybrid tracking scheme removes this contradiction: basic face points are computed locally, while facial expression refinement goes off-device only with a stable connection.

Below is how the modes compare by conditions of use and which mode limitations will have to be accepted in advance.

We choose the mode based on the device fleet profile and the real channel: we test some scenarios on weak models, not flagships — otherwise limitations will surface already in release.

Implementation stages: from audit to handover to the team

Face tracking implementation starts not with code, but with understanding how the app already works with the camera.

Engineers take measurements on real devices: where latency accumulates between frame capture and rendering, in which scenes the tracker loses the face, and how many resources processing takes from the rest of the interface.

1. Video processing audit. We break down the current frame path: capture mode, processing frequency, who releases buffers and when. The output is a list of bottlenecks with measurements, not assumptions.

2. Data flow scheme. We fix where the frame lives, where face points are computed, where smoothing runs, and how all of this is synchronized with rendering. The scheme becomes a contract between modules.

3. Model tuning. We select the tracking model and confidence thresholds for specific content: makeup, glasses, beard, headwear, low light. We configure smoothing separately so animation neither jitters nor lags.

4. Content-specific calibration. We run scenes from the product — masks, accessories, graphics overlays — and adjust the binding of points to the face. We observe behavior when the face is occluded by a hand and when leaving the frame.

5. Load testing. Long sessions, heat, weak devices, multiple faces in frame. We evaluate not average values but worst cases: where failures appear and how quickly the system returns to normal.

6. Handover to the team. We hand over configurations, the flow scheme, a set of test scenes, and metrics that reveal regression. We conduct training so the team can adjust calibrations itself.

What the client receives after implementation

Implementation can be considered complete not when the demo holds the face in frame reliably, but when your team can change parameters itself and understands what exactly will break.

That is why, along with working face tracking, we hand over a set of artifacts — from configurations to the policies by which the app lives after launch.

• Tracking configurations. Profiles for scenarios — close-up portrait, group shot, low light, motion capture; each with a description of which thresholds change and what that affects.
• Data flow schemes. The path from camera frame to rendered mask: where the stream is buffered, at what point graphics and recognition are synchronized, and where frames are lost during a drop.
• Data access rules. What stays on the device, what goes to the server, how long it is stored, who has access, and how it is logged.
• Model update policy. When a model can be replaced, how accuracy is checked before release, by which metrics degradation is caught, and how to roll back to the previous version.
• Integration documentation. Module map, extension points, list of events and exchange formats between app layers.
• Instructions for the team. Acceptance checklist, set of test scenes, performance thresholds, and the procedure for handling a drop on a weak device.

Case study: how we eliminated facial animation jerks in a virtual try-on app

The project was a virtual try-on app, MirrorFit: a virtual mask with the user’s facial expressions was overlaid on the face in real time. On flagship models it worked reliably; on the rest of the fleet it fell apart into jerks and animation dropouts.

What it was before

On a third of the device fleet, animation dropped out in the first seconds of the try-on: tracking lost the face on a fast head turn, and the contour did not recover — the mask remained frozen in the last pose.

In another one in five sessions, facial animation was jerky: facial morph target weights updated unevenly, and the face twitched on a smile. Users closed the try-on before they could examine the product.

What we changed

We split tracking into independent layers — face detection, head pose estimation, and facial expression estimation — and stopped updating them in a single pass: facial expression does not need the full pose rate.

We added pose prediction one frame ahead, weight smoothing with an adaptive window (short during sharp movements, long at rest), and a limit on active morph targets.

We replaced hard dropout with degradation: when the face is lost, the mask smoothly fades out and returns in fractions of a second, and the frame does not tear.

What became

The share of sessions with facial animation jerks fell from 22% to 3%, and animation dropout fell from a third of the device fleet to zero: failure moved into the class of degradation unnoticeable to the user.

Average try-on duration grew about twofold, and the share of repeat visits by a third. We keep smoothing thresholds and limits in configuration, so they can be adjusted without rebuilding the release.

What to do if face tracking is already implemented?

The first thing to do is not rewrite the solution, but break it into layers and understand exactly where stability is lost. Usually there are four independent areas: frame capture, face detection, face tracking itself, and passing the result to animation.

Until the cause is localized, any improvement turns into guesswork — the team fixes animation where capture is to blame.

Tracking diagnostics starts with telemetry on real devices, not on a test bench. We collect the share of frames with confident capture, recovery time after face loss, keypoint drift on a still head, behavior in low light, sharp turns, and partial occlusion. We compare configuration sets by these numbers — the bottleneck is immediately visible.

Some defects are closed by targeted component replacement, without rebuilding the entire pipeline. The detection model is changed to one more robust to angles, smoothing is retuned to remove jitter without lag, and reset and reacquisition rules are rewritten.

If drops come from lighting, working with exposure and frame preprocessing helps, rather than replacing the tracker.

Rebuilding the processing is needed when frames go through a single stream: latency accumulates, and on a weak device the solution falls apart entirely.

Then capture, computation, and rendering are separated, a buffer with discarded stale frames is introduced, and unnecessary memory allocations in the processing loop are removed. After that, targeted fixes become effective again and do not break everything else.

Technical questions asked before kickoff

Before kickoff, engineering teams ask roughly the same set of questions: will face tracking withstand real shooting conditions, will it integrate into an already released app, how will it behave on a weak device fleet, and who will own the deliverables. Below are short answers.

We assume that the face in frame is a moving object with highlights, occlusions, and inconsistent lighting, not a reference portrait from a test set.

Glasses, beard, medical mask?

Occluded zones are not reconstructed out of thin air — they are filled by symmetry and neighboring stable points, so corners do not “float” or twitch. If your audience has characteristic accessories, we fine-tune on your recordings.

Can it be added to a released app?

The component is delivered with a narrow interface: frame in, face parameters out — the app's screen and logic are not rewritten. We roll it out gradually and keep the ability to disable it remotely if behavior differs on some devices.

How do you test on weak devices?

We maintain a matrix of lower-price-segment devices and look at response time, share of frames with a detected face, and battery consumption. Under load, a lightweight mode turns on: mesh detail is lower, update rate is preserved so animation does not float.

Who owns the deliverables?

We transfer the code, model, and trained weights to you; we retain only the anonymized base components used before the project. Recordings for fine-tuning are stored separately and do not enter the general dataset.

Let's discuss the task and come back with an implementation plan

To come back with an estimate of the scope of work rather than generalities, we need inputs on the task. Below is the minimum from which the work scheme and stage order are assembled.

• Target devices and the weakest model in the support list — we select the tracking mode and behavior under load for it.
• Shooting scenario: one face in frame or several, distance, lighting, how far the head turns.
• What animation should output: head position, face morphs, character skeleton, or all together.
• How we check accuracy: on which recordings we count the share of frames with correct tracking and what we consider a dropout.
• Time budget per frame for tracking and rendering — it determines model choice and depth of post-processing.
• Privacy: on-device processing or sending frames to a server, whether video needs to be stored.
• What already exists: prototype, engine, test recordings, labeling — this shortens the first stage.

These inputs form the implementation plan: from a tracking prototype and measurements on your recordings to integration into the app and acceptance against pre-agreed metrics.

We separately discuss the points where we can stop and review the result, and what is included in each stage.

Describe the task in a convenient form — we will come back with an estimate, work scheme, and stage order.