Facial Emotion Recognition Implementation in Mobile Applications
Emotion recognition is face detection, extracting facial cues, and classifying into basic emotions (happiness, sadness, anger, surprise, fear, disgust, neutral per Ekman's model). Straightforward sounding, but industrial system accuracy in real conditions (varying light, partial occlusion, cultural differences in expression)—an active research topic.
What Works on Mobile
On iOS: VNDetectFaceLandmarksRequest provides 76 landmarks—enough to compute geometric descriptors (distance between mouth corners, eye opening degree, eyebrow angle). Train a small classifier on these descriptors as a CoreML model (MLP 3–4 layers). This approach is more stable than direct CNN on image, especially in poor lighting—landmarks are normalized to head position.
On Android: ML Kit Face Detection with setContourDetectionEnabled(true) gives 468 points—full face mesh. Excessive for emotion classification but enables precise facial muscle tracking.
Alternative: MediaPipe Face Landmarker—cross-platform, 478 landmarks + blendshapes (52 parameters like mouthSmileLeft, eyeBlinkRight, browDownLeft). Blendshapes are already semantic facial expression descriptors, feed directly to classifier without additional geometry. MediaPipe Face Landmarker latency on Pixel 7: ~15 ms.
Emotion Classification Models
Ready on-device options: HSEmotion TFLite (7 classes, ~4 MB), MobileNet-based emotion classifier (FER2013). Validation accuracy: 65–72% on 7 classes. In real conditions—lower. Not a bug but fundamental limitation: "neutral" and "thoughtful" expressions are extremely hard to classify.
For business cases (engagement analytics in edtech, measuring reaction to ad content), don't work with instantaneous classification. Use averaged values over 2–5 seconds and aggregated metrics: % time with positive emotion, % neutral, surprise spikes.
Reaction Animation
If the app reacts to user emotion (edtech mascot, interactive character), latency matters. Cycle: frame capture → inference → animation update must fit in 100 ms, otherwise reaction feels delayed.
On iOS: SwiftUI + withAnimation(.spring()) for smooth mascot state transition. Inference on background queue, result via @Published → @StateObject on main actor. On Android: Animator + MotionLayout for complex animation transitions.
Real case: educational app for kids with game elements. Character reacts to child's smile—dances if smile held >1.5 seconds. Used MediaPipe Face Landmarker + mouthSmileLeft/Right blendshape value > 0.6 as trigger. Problem: child laughs with open mouth—mouthOpen blendshape confused the filter. Added condition: mouthSmile > 0.6 AND mouthOpen < 0.4 OR (mouthOpen > 0.4 AND jawOpen > 0.3). False triggers reduced 40%.
Engagement Analytics
For A/B testing content (which screen triggers more positive reaction)—aggregate emotion scores per session, send to analytics. Data—not photos, only numeric vectors. User consent via explicit opt-in (emotion analytics—sensitive data).
Timeline
MediaPipe / ML Kit detection + custom classifier + reaction animation: 1–2 weeks. Engagement analytics dashboard: additional 1 week. Cost calculated individually.