Implementing AR Face Masks and Filters
Masks and filters are the most common face AR scenario. Snapchat, Instagram Reels, TikTok have conditioned users to a certain quality level. If mask "slides" when head turns or doesn't react to facial expression — user deletes app. Technically task is solved well by ARKit/RealityKit, but there are nuances with occlusion, animation, and video recording.
Types of Masks and Their Technical Implementation
Geometric mask — 3D mesh stretched over face following its deformations. Most realistic variant. In RealityKit: ModelEntity with skinned mesh, joints of which are tied to ARKit blendShapes:
// Face Anchor as root
let faceAnchor = AnchorEntity(.face)
// Load USDZ with morph targets matching ARKit blendShapes
let maskEntity = try! ModelEntity.load(named: "zombie_face.usdz")
faceAnchor.addChild(maskEntity)
arView.scene.addAnchor(faceAnchor)
// In update loop — apply blendShapes to mask morph targets
func session(_ session: ARSession, didUpdate anchors: [ARAnchor]) {
guard let faceAnchor = anchors.first as? ARFaceAnchor else { return }
// BlendShape mapping: ARKit jawOpen → mask jaw morph target
}
Flat sticker/overlay — 2D image or video tied to face anchor. Simpler to implement, fewer deformations. Good for frames, "hats", fake mustaches.
Particle effects — particles tied to face points (sparks from eyes, smoke from mouth when jawOpen > 0.3). In RealityKit — Entity with custom particle system via Reality Composer Pro.
Face Occluder: Mask Occludes Real Objects
Main technical nuance — virtual mask elements (horns, crown, glasses) must be occluded by real face where physically correct. Crown goes behind head — back of head occludes it. Glasses "sit" on nose — nose on top of frames.
In ARKit/RealityKit — occluder mesh: copy of face geometry with .occlusion material. Mesh invisible to user but writes depth to depth buffer. Elements "behind" face are properly occluded:
// Occluder entity — invisible copy of face
var occluderMaterial = OcclusionMaterial()
let occluderEntity = ModelEntity(
mesh: .init(arFaceGeometry: faceAnchor.geometry),
materials: [occluderMaterial]
)
faceAnchor.addChild(occluderEntity)
Without occluder horns stick through head from any angle — looks cheap.
Animation by Facial Expression
52 blendShapes ARKit → mapping to 3D mask morph targets. Standard scenario: zombie mask opens mouth on jawOpen, squints eyes on eyeBlinkLeft/Right. Mesh mask should have corresponding morph targets with same names or via mapping table.
Reality Composer Pro lets you set BlendShapeWeightsMapping directly in USDZ without code. For complex animations — custom update in ARSessionDelegate:
// Pass blendShapes to mask shader parameter
maskEntity.model?.materials[0].setParameter(
name: "jawOpen",
value: .float(faceAnchor.blendShapes[.jawOpen]?.floatValue ?? 0)
)
Recording Video with AR Mask
RPScreenRecorder (ReplayKit) — records entire screen including AR view. Simple, but low quality (records what's on screen, including UI elements). For clean AR camera recording — ARView.session + custom CVPixelBuffer render via Metal:
// Composite: camera frame + AR overlay → CVPixelBuffer
// → AVAssetWriter → .mp4
More complex, but gives clean recording without UI overlay and with choice of resolution.
Lens Studio / Spark AR as Alternative
If goal is viral filter for Snap/Instagram/TikTok, not own app: Lens Studio (Snap) and Meta Spark (Instagram) provide effect development environments without native code writing. Effects published on platform and available to users in native apps.
For custom mobile app with own camera — only native implementation or Banuba Face AR SDK.
Timeline
Basic mask (static 3D mesh, no blendShape animation) — 4–6 days. Animated mask with face occluder and 3–5 blendShape reactions — 2–3 weeks. Video recording + mask gallery with cloud upload — another 2–3 weeks. Cost calculated individually.