Real-time Video Filter Implementation in Mobile Apps
Developing video filters for mobile apps is always a trade-off between image quality and performance. One client came with a task: apply LUT filters to 4K 60fps video in real time. On an iPhone 14 Pro, initial tests delivered 12 fps—dropping every other frame. The issue was creating a new CIContext for each frame. After optimization, GPU utilization dropped to 30%, and fps stabilized at 60. We use GPU shaders for frame processing. On iOS: Swift with Metal; on Android: GLSL or Vulkan. For cross-platform projects, we implement native modules via platform channels.
Two Modes: Preview and Export
Real-time preview (during playback) requires GPU processing without writing the result to a file. Export—final recording with applied filter—can take seconds but must be frame-accurate.
Why a Single CIContext Matters
Creating a new CIContext per frame is a performance killer. On an iPhone 13, this yields 3–4 fps instead of 30. The context is created once and reused. A Metal-backed CIContext is mandatory: CIContext(mtlDevice: MTLCreateSystemDefaultDevice()!). Without explicit specification, some devices fall back to CPU rendering, causing video stutter.
How to Avoid FPS Drops During Export
The CVPixelBuffer from renderContext.newPixelBuffer() must be returned immediately after finish(withComposedVideoFrame:). Holding a reference leads to buffer pool exhaustion and crashes with kCVReturnInvalidArgument. On Android: GPUImageFilter is not thread-safe—you cannot apply the same instance from multiple threads simultaneously.
Platform Comparison
| Platform | Preview | Export | Tools |
|---|---|---|---|
| iOS | AVPlayer + Metal | AVVideoComposition + CIContext | Swift, Metal, Core Image |
| Android | GPUImageView + GLShader | media3 Transformer + GlEffect | Kotlin, GPUImage, media3 |
| Flutter | PlatformView (native code) | MethodChannel + native filter | Dart + Swift/Kotlin |
Performance: CPU vs GPU
| Approach | Throughput (60fps 1080p) | Latency |
|---|---|---|
| CPU (vImage) | 12–15 fps | ~70 ms |
| GPU (Metal/GL) | 60 fps | ~16 ms |
iOS: AVVideoComposition + Core Image
AVVideoCompositionCoreAnimationTool is suitable for static overlays like text and logos. For pixel-level filters, we use AVVideoComposition with a custom AVVideoCompositing:
class FilterCompositor: NSObject, AVVideoCompositing { func startRequest(_ asyncVideoCompositionRequest: AVAsynchronousVideoCompositionRequest) { guard let frame = asyncVideoCompositionRequest.sourceFrame(byTrackID: trackID) else { return } let ciImage = CIImage(cvPixelBuffer: frame) let filtered = applyFilter(ciImage) let output = asyncVideoCompositionRequest.renderContext.newPixelBuffer()! ciContext.render(filtered, to: output) asyncVideoCompositionRequest.finish(withComposedVideoFrame: output) } } For LUT filters, we use CIColorCubeWithColorSpace with a 64×64×64 table. Learn more about Core Image.
Android: GPUImage and media3 Effects
Real-time preview uses GPUImageView with a custom GPUImageFilter. Shaders are written in GLSL:
precision mediump float; uniform sampler2D inputImageTexture; varying vec2 textureCoordinate; void main() { vec4 color = texture2D(inputImageTexture, textureCoordinate); float luma = dot(color.rgb, vec3(0.299, 0.587, 0.114)); gl_FragColor = vec4(mix(vec3(luma), color.rgb, 1.3), color.a); } For export with filter, we use media3 Transformer with GlEffect. MatrixTextureProcessor accepts the shader and applies it to each frame during transcoding.
Flutter. Full real-time video filters require platform code. Native plugins with MethodChannel are the only viable approach for production applications.
Applying Filters to Live Camera Recording
An even more demanding scenario is applying filters during capture. On iOS: AVCaptureSession + AVCaptureVideoDataOutput → SampleBufferDelegate → Metal shader → render into MTKView. The chain latency must be < 16 ms for 60fps. The pixel format kCVPixelFormatType_420YpCbCr8BiPlanarFullRange (YUV) converts faster to a Metal texture than BGRA.
On Android: CameraX Analysis + ImageAnalysis.Analyzer → GLES shader → GLSurfaceView. Using ImageAnalysis.STRATEGY_KEEP_ONLY_LATEST discards old frames if the GPU hasn't finished processing the previous one.
Process of Work
- Analysis: Assess filter requirements, target devices, desired fps.
- Design: Choose architecture (native/cross-platform), prototype shaders.
- Implementation: Write GPU shaders, integrate with AVFoundation/Media3, optimize performance.
- Testing: On real devices (iPhone 8–14, Samsung Galaxy S10–S23), measure fps, check for artifacts.
- Deployment: Publish to App Store/Google Play, configure TestFlight/App Distribution.
Common Export Mistake
Often, developers forget to reuse `CIContext`—resulting in fps dropping to 2–3. Also, returning `CVPixelBuffer` immediately after use is crucial; otherwise, the buffer pool gets exhausted.What Is Included in the Work
- Source code for shaders and integration (Swift, Kotlin, Dart).
- Integration and configuration documentation.
- Instructions for adding new filters.
- One month of support after delivery.
- Testing on 3+ real devices.
Timeline and Guarantees
5–7 business days: Develop shaders for 5–8 filters, integrate preview and export on iOS and Android. Flutter version with native code adds 2 days. The cost is calculated individually—contact us for an estimate. We guarantee stable filter operation on devices from iPhone 8 and Android 10. Our experience: 5+ years in mobile development, 30+ video-processing projects. Get in touch—we'll help implement video filters for your app.