Real-Time AI Video Segmentation for Mobile Apps
Imagine a video call with a blurred background, but instead of a smooth picture—stuttering and artifacts. Or an augmented reality app that can't keep up with user movements. These are typical consequences of poorly optimized AI segmentation. We often receive requests for real-time video segmentation—when the app understands what's in the frame: person, background, car, road, and does it for every frame at 15–30 FPS. Making it "work on a demo" is easy. Making it "doesn't overheat, doesn't lag, works on iPhone XR" requires serious optimization. Choosing the right model and a smart processing pipeline are key to a successful project.
How to Choose a Segmentation Model?
Semantic segmentation—each pixel is assigned a class (background, person, car). Applications: background replacement in video calls, AR effects, road scene analysis. Instance segmentation—a separate mask for each object of the same class (three cars—three masks). Applications: object counting, tracking. Panoptic segmentation—a combination. Heavier, rarely used on mobile.
| Model | Resolution | FPS (CoreML) | Quality |
|---|---|---|---|
| MobileNetV3-DeepLabV3 | 513×513 | 22–28 | Medium |
| EfficientPS-lite | 640×360 | 18–24 | Good |
| YOLOv8n-seg | 640×640 | 20–30 | Good |
| Segment Anything (SAM-mobile) | 1024×1024 | 3–5 | Excellent |
YOLOv8n-seg is 2x better in quality than MobileNetV3 at similar speed. SAM is only for interactive segmentation, not real-time.
Why Is Performance Critical?
Mobile GPUs are limited in heat dissipation and power consumption. A naive implementation quickly leads to overheating and FPS drops. Optimization starts with model selection: lightweight architectures (MobileNetV3, YOLOv8n-seg) give 20–30 FPS on modern chips, but require a smart pipeline. According to Core ML documentation, proper Metal render configuration reduces CPU load by 40%.
How We Optimize the iOS Pipeline
class RealtimeSegmentationProcessor { private let model: VNCoreMLModel private let processQueue = DispatchQueue(label: "segmentation.process", qos: .userInteractive) // Frame skipping: process every N-th frame private var frameCounter = 0 private let processEveryNFrames = 2 // 30fps camera → 15fps processing func captureOutput(_ output: AVCaptureOutput, didOutput sampleBuffer: CMSampleBuffer, from connection: AVCaptureConnection) { frameCounter += 1 guard frameCounter % processEveryNFrames == 0 else { return } guard let pixelBuffer = CMSampleBufferGetImageBuffer(sampleBuffer) else { return } processQueue.async { [weak self] in self?.runSegmentation(on: pixelBuffer) } } private func runSegmentation(on pixelBuffer: CVPixelBuffer) { let request = VNCoreMLRequest(model: model) { [weak self] req, _ in guard let observation = req.results?.first as? VNCoreMLFeatureValueObservation, let maskArray = observation.featureValue.multiArrayValue else { return } let mask = self?.processMask(maskArray) DispatchQueue.main.async { self?.delegate?.didUpdateSegmentationMask(mask) } } // Important: pixelBuffer must be in the correct format request.imageCropAndScaleOption = .scaleFill let handler = VNImageRequestHandler(cvPixelBuffer: pixelBuffer, orientation: .right) // landscape orientation try? handler.perform([request]) } private func processMask(_ array: MLMultiArray) -> SegmentationMask { // Convert MLMultiArray → CVPixelBuffer for rendering // Shape: [numClasses, height, width] let numClasses = array.shape[0].intValue let height = array.shape[1].intValue let width = array.shape[2].intValue // Argmax across classes for each pixel → label map var labelMap = [UInt8](repeating: 0, count: height * width) for y in 0..<height { for x in 0..<width { var maxClass = 0 var maxVal: Float = -Float.infinity for c in 0..<numClasses { let val = array[[c, y, x] as [NSNumber]].floatValue if val > maxVal { maxVal = val; maxClass = c } } labelMap[y * width + x] = UInt8(maxClass) } } return SegmentationMask(labels: labelMap, width: width, height: height, classColors: Self.classColorMap) } } What Is Metal Mask Rendering?
Naive approach—drawing the mask in a CPU loop—gives 3–5 FPS on rendering. The right way is Metal / OpenGL ES. A Metal shader applies the mask on the GPU without CPU involvement, ensuring stable 30 FPS even on iPhone 11.
// Metal shader for overlaying mask on video // Inputs: videoTexture (YCbCr), maskTexture (label map), colorLUT (class→color) fragment float4 segmentationOverlay( VertexOut in [[stage_in]], texture2d<float> videoTexture [[texture(0)]], texture2d<uint> maskTexture [[texture(1)]], texture1d<float> colorLUT [[texture(2)]], constant OverlayParams& params [[buffer(0)]] ) { float2 uv = in.texCoords; float4 videoColor = videoTexture.sample(sampler, uv); uint classLabel = maskTexture.sample(nearestSampler, uv).r; if (classLabel == 0) { return videoColor; } // background—unchanged float4 maskColor = colorLUT.sample(sampler, float(classLabel) / float(params.numClasses)); return mix(videoColor, maskColor, params.overlayAlpha); // blending } | Platform | Rendering Engine | FPS (mid-range devices) |
|---|---|---|
| iOS (Metal) | GPU compute + shader | 28–30 |
| Android (MediaPipe+OpenGL) | GPU delegate + overlay | 25–30 |
Background Replacement on Android: MediaPipe
For video calls, binary segmentation (person/background) is popular. We use MediaPipe Selfie Segmentation—a ready-made solution optimized for this task:
// Android: MediaPipe Selfie Segmentation val options = ImageSegmenterOptions.builder() .setBaseOptions(BaseOptions.builder() .setModelAssetPath("selfie_segmentation.tflite") .setDelegate(Delegate.GPU) .build()) .setRunningMode(RunningMode.LIVE_STREAM) .setResultListener { result, _ -> val confidenceMask = result.confidenceMasks?.get(0) updateBackground(confidenceMask) } .build() val segmenter = ImageSegmenter.createFromOptions(context, options) Delegate.GPU is critical: on CPU, the same MediaPipe delivers 8–12 FPS; on GPU, 25–30 FPS.
Workflow: From Idea to Production
- Task analysis—define object classes, target devices, FPS requirements.
- Model selection—test 3–5 models on real devices, benchmark speed and mIoU.
- Pipeline optimization—configure frame skipping, Metal rendering, TFLite GPU delegate.
- Integration—embed into the app, error handling (graceful fallback to CPU under throttling).
- QA and deployment—stress testing, App Store / Google Play release.
What's Included
- Model selection and calibration for your data.
- Implementation of the capture and frame processing pipeline.
- GPU mask rendering (Metal on iOS, OpenGL/Vulkan on Android).
- Integration documentation and support.
- 30-day post-release support.
Timelines and Cost
Basic single-class segmentation (e.g., person) with a ready-made model—from 1 week. Custom multi-class segmentation with Metal/GPU rendering and iOS + Android support—from 2 to 4 weeks. Cost is calculated individually after evaluating your project.
We have 5 years of mobile development experience and over 50 AI-powered projects. We guarantee stable 30 FPS on devices not older than iPhone XR and Samsung S10. Contact us for a technical consultation and accurate estimate. Request a pilot project to evaluate quality on real data.