AI Virtual Glasses Try-On in Mobile Apps
Warby Parker, EyeBuyDirect, Zenni Optical — all major online opticians deployed AR try-on. Conversion to purchase is higher for users who tried glasses virtually. Technically this is one of the most accurate AR try-on scenarios: glasses are rigid objects with known geometry, attachment points on face (nose, temples) are fixed and well-defined by face tracking.
Face Tracking and Glasses Placement
ARKit (iOS, TrueDepth). ARFaceAnchor.geometry provides depth-accurate face mesh. Key points for glasses: nose bridge landmarks + temple landmarks. ARKit provides blend shapes — use eyeBlinkLeft/eyeBlinkRight for animation: when user blinks, glasses stay in place (obviously), but this allows correct rendering during blink without artifacts.
Positioning: take ARFaceAnchor.transform as world transform → place 3D glasses model relative to face anchor with offset by nose bridge point. Interpupillary distance (IPD) from face geometry (leftEyeTransform, rightEyeTransform) → scale glasses model to specific face size.
MediaPipe Face Landmarker (Android / cross-platform). 478 points including irises. Points 1, 2, 98, 327 — nose bridge; points 234, 454 — temples. Head transformation matrix via Facial Transformation Matrix from MediaPipe. On Android — inference via TFLite or native MediaPipe Tasks API.
3D Glasses Models: Requirements
Frame catalog — 3D models in glTF 2.0 (PBR materials). Requirements:
- Accurate 1:1 scale in meters (standard frame ~140 mm temple-to-temple)
- Model origin at nose bridge center — anchor point to face anchor
- Materials: metal (
metallic: 1.0, roughness: 0.1), plastic (metallic: 0.0, roughness: 0.6), gradient tint for lenses - LOD: for real-time sufficient 2000–5000 polygons per frame
- Lenses — separate mesh with transparent/tinted material simulating glass
Conversion from frame suppliers: often receive OBJ or FBX without PBR materials. Pipeline: Blender → retopology if needed → PBR material assignment → glTF export → validate via gltf-validator.
Lenses: Transparency and Tint
Lenses in AR are complex materials:
- Transparency: alpha blending,
SCNMaterial.transparency - Tint (sunglasses): colored transparent material — HSV overlay over background capture
- Anti-glare coating: subtle specular highlight on lens outer surface
- Photochromic: animated tint change based on
ARLightEstimate.ambientIntensity
On ARKit: render camera background → face mesh as occluder → glasses on top. Lenses — separate pass with additive blending to "see through" to real world.
Size Compatibility and Recommendations
Useful feature: determine if frame fits face size. From face geometry compute face width in real centimeters (temple-to-temple distance via simdDistance). Compare with catalog frame_width parameter. Show: "Frame 140 mm, your face width 145 mm — this frame will fit."
E-commerce Integration
AR try-on embeds in product detail screen. After try-on — direct "Add to Cart" button with selected size and color. Deep link to Safari/Chrome for WebAR (Quick Look for iOS, model-viewer for Android) as fallback for app-less users.
Analytics: which frames tried most, after which frames purchased — send try_on_started, try_on_duration, added_to_cart_after_tryon to Firebase.
Timeline: AR glasses try-on with frame catalog for iOS: 5–8 weeks. Cross-platform iOS + Android with full PBR rendering, tinted lenses, and store integration: 10–16 weeks. Cost calculated individually.