Real-Time AI Virtual Makeup Try-On System

Virtual makeup try-on: where real time breaks

The virtual makeup try-on in the interface looks like a single button: turn on the camera — and there's lipstick on your face. Inside, it's a pipeline of four stages — finding the face, tracking key points, applying a shade layer, and sending the frame to the screen.

As long as the stages fit within the time budget of one frame, the image reads like a mirror; as soon as one falls out, it reads like a delayed filter.

The contour is the first to fall apart. Key points jump between frames, the mask "breathes" at the border of the lips and eyelids, and this is noticeable without zooming in.

Smoothing over frame history removes jitter but adds inertia: the face has already turned, while the contour is still catching up.

Response lag in face tracking is felt most strongly on movement — head tilt, smile, turn toward a light source.

If the stages are computed sequentially, the camera manages to output several new frames while the previous one is being processed, and desynchronization appears between the gesture and the reaction. Parallelizing the stages and a separate lightweight path for fast movements help here.

The third source is shade. A translucent layer is mixed with the base skin tone, so the same pigment drifts on different people and under different lighting: warm light pushes lipstick toward orange, cool light toward lilac.

Color creeping along the contour when lighting changes is the most common complaint about how real-time virtual makeup works, and the main blocker for selecting cosmetic shades online: the customer does not trust a color that changes along with the lamp in the room.

We break down how this is closed: where normalization against a reference skin image is needed, and where catalog calibration to real pigments is needed.

Application accuracy and face tracking stability

Try-on quality stops being a matter of taste: we hold four measurable indicators — deviation of the lip and eyelid contour from the real boundaries of the face, tracking stability during head rotation, the share of frames with face loss, and the color discrepancy between the applied shade and the reference.

We measure not on studio photos, but on recordings from ordinary cameras — the ones the user will actually work with.

Tracking stability rests on a dense facial point mesh and temporal smoothing: the contour position between frames is interpolated, so makeup does not jitter on the lips during facial expressions and does not "float" on the cheekbones.

The base layer is MediaPipe Face Mesh, with our own filters and frame-by-frame stabilization on top.

When the face is occluded by a hand, hair, or eyeglass frames, the mask does not drift: it freezes in the last reliable position and returns to computation as soon as the area is visible again.

Low lighting is a separate scenario. Before computation, the frame is normalized by brightness and noise, and the tracking confidence threshold changes in a controlled way so that false positives on shadows do not appear.

On the test set, the share of frames with an uncertain contour drops from double-digit values to tenths of a percent — try-on remains functional under desk lamp light and in an evening room.

We test on real shooting scenarios: backlight from a window, a lamp above a desk, evening lighting, head rotations to the limit, active facial expressions, hands near the face.

Each scenario is a separate run with frame-by-frame annotation, so what is visible is not abstract "smoothness" but specifics: frame stability, share of tracking failures, color discrepancy of the shade across different skin tones.

Based on these numbers, the client accepts the work, and we record regressions during subsequent model updates.

Launch options: browser, app, mirror at the point of sale

The same makeup try-on algorithm behaves differently depending on the entry point. A browser, a mobile app, and a mirror with a camera at the point of sale differ in camera access, channel requirements, and where the model and shade matching are actually computed.

The second decision is where computation happens: on the visitor's device or on the server. It affects privacy, image stability, and how quickly we can update palettes and matching rules.

In practice, a hybrid is more often chosen: a lightweight model runs on the device for fast feedback, while complex scenes and precise shade matching go to the server.

Before launch, we run both modes on your scenario and record which one delivers the required smoothness without compromising privacy.

Implementation stages: from task analysis to system handover

The order of work is arranged so that contentious parts of the scene — flash glare, cold store-window light, dark skin tone — surface on the test bench, not on handover day. Next is the route from the first task analysis to a working system.

1. Audit of shooting scenarios. We collect real conditions: a phone's front camera or webcam, window or store-window light, close-up, head rotations, smile. For each scenario, we record what the system must withstand and where it is allowed to refuse a result instead of producing a false overlay.
2. Solution design. We agree on facial areas for tracking, anchor points for application, and layer order — tone, blush, eyeshadow, lips — as well as how makeup behaves during fast movement.
3. Preparation of reference sets. We annotate frames by skin tone, lighting type, and age. The set is fixed so that iterations are compared against the same data, not "by eye."
4. Setup of palettes and application rules. We bring shades to a predictable result under different lighting: mixing with skin tone, density, behavior on highlights and on dark areas.
5. Acceptance checks. We run control scenarios: share of frames without application failure, response to movement, stability on weak devices. Discrepancies are resolved before handover, not after launch.
6. Handover to operation. We provide configurations, documentation on palettes and rules, instructions for adding new shades, and train the client's team.

This sequence leaves behind a system that can be developed in-house: new shades and scenarios are added through configuration, without rewriting the foundation.

What we hand over: configurations, schemes, and regulations

After delivery, your team retains not only a working service but also a set of documents by which it can be developed without us.

Below is exactly what is included in the handover and why each artifact is needed in practice: from editing a single palette to connecting a new storefront.

• Configuration set — face detection thresholds, application points, and rendering parameters are moved into separate files rather than hardcoded: shades and behavior can be adjusted without rebuilding and releasing.
• Integration schemes with the website and app — call order, request and response formats, behavior when the service is unavailable, and degradation to a static preview.
• Image access policies — where frames are stored, how long they live, who requests them and on what basis, and what goes into analytics. This closes security service and personal data requirements.
• Palette update regulation — how to add a shade, calibrate it for different lighting, and roll it out to production without breaking already configured sets.
• Reference frames and quality metrics — a set of test scenes against which every new palette and model update is checked before rollout: it shows whether things got better or worse.
• Deployment and rollback instructions — environments, variables, rollout order, and return to the previous version in one step.
• Extension point catalog — where to connect new product categories and which modules are affected.

Case study: how we removed freezes and sped up response

The try-on implementation case began with the cosmetics retail chain Lumé: shade try-on worked directly in the product card, shoppers used it, but only a portion reached purchase.

The tone layer appeared with a noticeable pause after head rotation, the mask "slid" during movement and lighting changes, and on some smartphones the session broke off midway.

According to logs, 4 out of 10 shoppers abandoned the try-on, and almost none of them returned to the product card.

What the analysis showed

The bottleneck was not the application algorithm itself, but the frame path: compression, sending to the server, waiting for a response, and rendering back. The weaker the channel and the more loaded the device, the longer this chain became — hence the freezes and drop-offs.

Measurements on real devices produced pre-change metrics: try-on response to head movement — about a second, share of interrupted sessions — 23%, share of transitions from try-on to cart — 6%.

What we changed and what happened

We moved application computation to the shopper's device and reworked facial annotation so that it relied on stable points and did not "breathe" between frames.

We added controlled degradation: with a weak channel, try-on reduces shade detail but does not break the session, and we keep palettes on the device — switching between tones stopped being a separate request.

The acceleration of try-on response was multiple-fold: the layer reacts to head movement without a pause noticeable to the eye. The share of interrupted sessions fell from 23% to 4%, cart transitions rose from 6% to 14%, and the load on try-on servers decreased almost sevenfold.

The failure class "interface froze and lost the face in frame" disappeared entirely — instead, the shopper sees only a soft reduction in image quality, which they do not notice.

How is try-on different from applying a filter?

A filter mask is a ready-made translucent texture glued to the face at several anchor points. It knows neither skin tone nor lip relief: the shade is predefined, so on dark skin, in side light, or on lip gloss it looks like a sticker. Hence the characteristic "mask" effect — there is color, but no cosmetics.

Try-on is built differently: first geometry and color, then pigment. We reconstruct a dense facial mesh, segment the areas of the lips, eyelids, and skin, estimate tone and lighting from reference areas of the frame — and only then transfer the shade.

Lips and eyes are the hardest part of transfer: there is relief, highlights, translucent transitions, and a delicate border with the skin.

The shade is applied taking the lighting of the frame into account, while gloss and texture are preserved — this is exactly why lipstick looks applied rather than drawn.

The limitation of a filter is that it does not distinguish occlusions: hair, glasses, a hand near the face do not exist for it. Plus, shadows on the mask do not match the light in the frame. For entertainment this is enough; for choosing a cosmetic shade, it is not.

Technical doubts before launch: we answer substantively

At the contractor selection stage, the questions are almost always the same: will the service withstand the flow of shoppers in the hall, what will owners of inexpensive smartphones see, will try-on survive a sale day, and where do photos of faces go. Below are direct answers based on the design of the pipeline, not on promises.

Will try-on work on an inexpensive smartphone?

The model is supplied in two branches of different weight. The lightweight one maintains a smooth image on devices from previous years without overheating the body, and the heavy one turns on only where there is performance headroom.

If the device does not pass the threshold at all, we show try-on on an uploaded photo — either way, the shopper sees the shade on themselves, not an empty screen.

What happens during peak traffic on a sale day?

The computation pipeline scales horizontally: as the number of simultaneous try-ons grows, compute nodes are added, and the queue prioritizes already started ones.

Before launch, we run a profile with a tenfold spike — under overload, the layer refresh rate decreases, but the service remains available and does not bring down other store sections.

Let's discuss the task: what to prepare for the first conversation

To come back with a plan and an estimate of work rather than general questions, we need the context of the task. The first conversation usually takes little time, and its usefulness depends on how precisely you describe where try-on works and what is considered a result. Below is what to prepare.

• Try-on use scenarios: a product card in an app, a screen in the sales area, a promo page, or a camera at a store window. This determines whether we build a separate module or integrate into an existing interface.
• Product catalog: photos of lipsticks, eyeshadows, and foundation shades, their codes, and grouping by lines. Clean source assets and a unified format noticeably reduce manual setup.
• Requirements for application accuracy: skin types, low lighting, movement of a person in frame, glasses, and hair covering the face. What we consider acceptable, and where try-on must honestly report that the face was not recognized.
• Brand rules: palette, typography, product presentation, and interface hints. Try-on should look like part of your product, not someone else's block on top of it.
• Technology stack and constraints: app or website platform, language, where the catalog is stored, how the module will connect to accounting systems and analytics.
• Acceptance criteria and responsible parties: who on the business and IT side accepts the result, what we consider a finished module, and which usage reports are needed.

Describe the task in free form — we will come back with an estimate and work plan. If some data is not available yet, come with what you have: together we will sketch out the structure and say which materials are critical and what can be clarified later.