Why on-device inference wins for real-time face aging preview
Rendering face aging or rejuvenation effects with sub-100ms latency on a mobile device is a tough technical challenge. Server-side models like SAM2 and StyleGAN deliver photorealistic results but take 2–20 seconds per image — unacceptable for interactive sliders. Our projects leverage specialized on-device models such as FRAN (Face Re-Aging Network), which are distilled into compact versions that run directly on the smartphone. FaceApp historically pioneered this on-device approach. We combine FRAN with CoreML on iOS and TensorFlow Lite on Android, ensuring instantaneous UI feedback. Our team has 10+ years in mobile AI development and has delivered 40+ projects with on-device neural networks.
Problems we solve
Latency vs. quality trade-off
The most common pain point is choosing between instant preview and high fidelity. On-device inference on an iPhone 13+ Neural Engine takes 60–90 ms for a 256×256 input. This enables live preview as the user drags the age slider. On older devices like iPhone X (A11 Bionic), we still achieve ~200 ms, acceptable with a 150 ms debounce. The alternative — server-only processing — forces a 2–20 second wait, breaking the interactive experience.
Alignment artifacts
Without precise face alignment, FRAN produces artifacts at head rotations beyond 15°. Standard pipelines using VNDetectFaceLandmarksRequest (76 points on iOS) or MediaPipe Face Mesh (468 points on Android) compute an affine transform from five key landmarks (eyes, nose, mouth corners). Warp via vImage on iOS or OpenCV on Android, then after inference reverse-transform and blend with Poisson blending.
Privacy concerns
Users hesitate to upload personal photos to the cloud. On-device processing guarantees 100% privacy — no data leaves the device. When server processing is required (for full-resolution exports), we delete original photos immediately after inference and clearly disclose this in the Privacy Nutrition Label.
How we do it: FRAN on CoreML
FRAN is an open-source model by Netflix Research trained on synthetic data. It takes a face image and a target age (normalized 0–1) and returns a stylized face. The CoreML version weighs ~45 MB in FLOAT16.
import CoreML import Vision class FaceAgingProcessor { private let model: FRAN func process(faceImage: CGImage, targetAge: Int) async throws -> CGImage { // FRAN accepts normalized 256x256 image let resized = try resize(image: faceImage, to: CGSize(width: 256, height: 256)) let input = FRANInput( face_image: try MLMultiArray(from: resized), target_age: MLMultiArray([Float(targetAge) / 100.0]) // normalize to 0..1 ) let output = try await model.prediction(input: input) return try cgImage(from: output.output_face) } } On recent projects, we have consistently achieved inference times between 60–90 ms on iPhone 13 and newer. The alignment pipeline adds ~10 ms, and Poisson blending (using Accelerate Framework on iOS or a Metal shader) adds another 20 ms. Total pipeline latency stays under 120 ms — ideal for real-time sliders.
Poisson blending for seamless transitions
Standard CIBlendWithMask leaves a hard mask edge. We implement Poisson Image Editing to blend the transformed face region smoothly. On iOS, this is done via Accelerate Framework by solving a system of linear equations, or via a custom Metal compute shader for better performance.
Process: from audit to launch
- Data collection — we analyze your target devices and performance requirements.
- Audit — evaluate model candidates (FRAN, SAM, or custom) against your quality bar.
- Design — choose on-device, hybrid, or server-only architecture.
- Estimation — provide a detailed effort and cost breakdown.
- Development — model conversion, pipeline implementation (detection, alignment, inference, blending), UI components (slider, animation), server endpoints (if hybrid).
- Testing — performance benchmarks on device farm, visual quality validation.
- Launch — assist with App Store and Google Play submission, including privacy labels.
What's included in the integration
- Model selection and optimization (FRAN, SAM, or custom)
- Conversion to CoreML / TensorFlow Lite with quantization
- Full pipeline: detection → alignment → inference → blending
- UI element: age slider with animated transition
- Server part: endpoint for upload/download (if hybrid)
- Integration with Photo Library and sharing system
- API and configuration documentation
- Store submission support (App Store Review, Google Play)
Timeline and cost
On-device integration with FRAN takes 5–8 days. The hybrid mode (on-device preview + server export) requires 2–3 weeks. Cost is determined after requirements clarification. Contact us for a project assessment — we will propose the optimal solution.
Common mistakes to avoid
- Skipping alignment — always apply affine warp for robust results.
- Using server-only for preview — users expect instant feedback; on-device is mandatory for sliders.
- Ignoring privacy labels — if photos leave the device, you must disclose data handling in the App Store.
- Missing fallback — on older devices, fallback to lower resolution or server when Neural Engine is unavailable.
The winning combination: on-device preview + server export
The best user experience is an instant on-device preview at 256×256 while dragging the slider, and a "Save" button that triggers server-side processing at original resolution. During server processing, show an animation. On completion, save the result to Camera Roll via PHPhotoLibrary. On Android, use WorkManager so the request persists even if the app is backgrounded, and notify the user when done.
Contact us to evaluate your project and choose the optimal path.