Implementing AR Glasses and Accessories Try-On
Glasses — one of best candidates for AR try-on: fixed shape, attaches to specific face points (bridge, ears) and doesn't require mesh deformation. But "well-fitting" glasses in AR — correct pivot, right scale for user IPD, and stable animation on head movement. Without these details, try-on frustrates rather than helps.
ARKit Face Tracking as Foundation
ARFaceTrackingConfiguration — only option for face AR on iOS. Requires iPhone X or newer (TrueDepth camera). Returns ARFaceAnchor with:
-
transform— face position and orientation in world coordinates -
geometry— 1220-point face mesh -
blendShapes— 52 coefficients for expressions (blink, smile, brow raise)
For glasses, use specific face mesh points: bridge (~vertex 9 in ARKit mesh), temples. Glasses model pivot point — strictly on bridge. With correct pivot, frame "sits" on face without extra position corrections.
// Bind glasses model to face anchor
let faceAnchorEntity = AnchorEntity(.face)
let glassesEntity = try! ModelEntity.load(named: "glasses.usdz")
glassesEntity.position = SIMD3(0, 0.005, 0.065) // fine Y, Z correction
faceAnchorEntity.addChild(glassesEntity)
arView.scene.addAnchor(faceAnchorEntity)
Scaling for Real Face Size
Standard glasses model in USDZ — for "average" face. Real faces differ. Two approaches:
Automatic. Measure face width via distance between left and right ear in ARFaceAnchor.geometry. ARKit gives these points in meters — compare with glasses frame reference width, compute scale coefficient. Apply to ModelEntity.scale.
Manual. Size slider in UI. Simpler to implement, worse UX.
Distance between pupils (IPD) via eye tracking available on iPad Pro (ARKit, ARFaceAnchor.leftEyeTransform, rightEyeTransform). On iPhone, face tracking doesn't directly measure IPD, but approximately computed from face geometry.
Headphones, Earrings, Hats
Same principles, different anchor points:
Headphones/Earrings → leftEarTransform, rightEarTransform from ARFaceAnchor. ARKit provides these transforms directly.
Hats → ARFaceAnchor.transform (head center) + Y displacement upward. Must account for head tilt — else hat "slides" when head tilts back.
Sunglasses with Lenses. Lenses with tinted/gradient material — via PBR with transmission and tint color. In RealityKit — .PhysicallyBasedMaterial with opacityThreshold and blending. Mirror lenses — environment texture reflection.
Android: Face Tracking Without TrueDepth
ARCore AugmentedFace (Pixel 3+, Samsung Galaxy S10+, others with depth sensor) — ARKit face tracking analog, but with lower mesh accuracy (468-point MediaPipe vs 1220 ARKit). For glasses sufficient — key bridge and temple points stable.
Devices without depth sensor: MediaPipe Face Landmark via ML Kit. Works on RGB camera, lower accuracy on movement, but for static frame — acceptable.
Performance and Memory
On iPhone older than XR, simultaneous face tracking + high-polygon model + environment texturing stresses GPU. Metal frames with reflection require environment probe — additional draw calls.
Optimization: LOD (level of detail) for glasses — full model on static frame, simplified on movement. 512×512 textures for fine frame details — sufficient for mobile screen.
Timeline
Face tracking + basic AR try-on single glasses model — 5-7 days. Gallery of frames, face scaling, sharing — 2-3 weeks. Android support via ARCore AugmentedFace — plus 1 week. Cost calculated individually.