Building a Virtual Fitting Room: AR Try-On Implementation

Building a Virtual Fitting Room: AR Try-On Implementation

We develop AR try-on for mobile applications — from simple accessories to full clothing with mesh deformation. The core problem we solve: realism. Without accurate pose tracking, scaling to user parameters, and physically correct lighting, the try-on becomes a "cardboard" collage and fails to boost conversion.

Our team integrates native frameworks — ARKit (iOS) and ARCore (Android) — as well as alternative SDKs to accelerate time-to-market. Each product category requires a specific technology: glasses need face tracking, clothing needs body tracking with mesh deformation, shoes require foot tracking or plane detection, and accessories need hand tracking.

The term "try-on" is an umbrella that breaks down into fundamentally different technical tasks. For each product category, a dedicated approach is needed: glasses and jewelry — face tracking (ARKit/ARCore), clothing — body tracking with mesh deformation, shoes — foot tracking or plane detection, hand accessories — hand tracking. The table below summarizes categories, technologies, and complexity estimates.

The most complex category is clothing. Without correct mesh deformation based on body pose, the clothing looks like a cardboard silhouette over the person.

How to Achieve Realistic Mesh Deformation for Clothing?

iOS: ARBodyTrackingConfiguration (A12+, iOS 13+) provides a 91-point skeleton in world coordinates. ARSkeleton3D with jointModelTransforms gives matrices for each joint. A skinned mesh of clothing is overlaid on the skeleton: each vertex is bound to 1-4 joints with skinning weights. When a joint moves, vertices follow according to the weights.

The skinned mesh format is USDZ with a SkinningComponent in RealityKit. Mesh preparation in Blender with rigging to the standard ARKit skeleton is a critical step. If the artist is not familiar with ARKit joint hierarchy, the rib cage and spine will deform incorrectly. More details can be found in Apple documentation.

Android: ARCore does not provide body tracking out of the box. Options include MediaPipe Pose (Google), BlazePose (33 keypoints), MoveNet (Google). These are not ARKit-level accuracy, but sufficient for a marketing try-on. Mesh deformation is handled via custom OpenGL/Vulkan shaders or TensorFlow Lite with pose estimation.

Why Lighting Is Critical for AR Try-On?

PBR material with correct roughness/metallic values plus ARKit environment probe (AREnvironmentProbeAnchor) or automaticEnvironmentTexturing is the minimum for realistic appearance. Without an environment map, the shiny leather of a bag looks identical to the matte fabric of a jacket.

Shadows from virtual clothing onto the real body require an occluder mesh: an invisible body model casts shadows onto the real floor/walls. In RealityKit, use the .occlusion material on an occluder entity. This greatly enhances realism but adds complexity (an accurate 3D body model of the user is needed).

Comparison of Third-Party SDKs

For a quick launch, ready-made SDKs can be used, but for a premium experience, a custom implementation with full control over deformation and lighting is better. Custom solutions typically outperform SDKs by 30-40% in user engagement and conversion.

Scaling to the User

Glasses are tried on without entering parameters — face tracking keeps them precisely on the nose. Clothing is different. The basic approach uses standard sizes S/M/L with mesh scaling based on the skeleton's shoulder bounding box. Accurate fitting with chest/waist circumference requires user input or body scanning (a separate task with even higher complexity).

What Our Work Includes