An AR app must perform AR environment recognition in milliseconds: where the floor is, where the walls are, what objects are in the frame, and how light falls. Without this, a virtual object 'floats' in the air, casts no shadows, and looks unnatural. The term Scene Understanding covers the foundation on which all AR experiences are built. Our team has over 10 years of proven experience in computer vision AR, with certified expertise and 50+ projects delivered. We've seen projects where occlusion ruined the UX — costly mistakes at the prototyping stage. Financial risks from incorrect scene understanding can reach 60% of an AR project's budget.
Scene understanding and its importance
Scene understanding combines several CV tasks: plane detection, depth estimation, semantic segmentation, and light estimation. Each is critical for realistic AR. For example, without correct depth estimation AR, occlusion fails — a virtual object cannot hide behind real objects. Compare: LiDAR provides depth with ±1–2 cm accuracy, while monocular depth gives ±5–10 cm, 5 times worse at close range. Our custom depth model is 2.5x more accurate than built-in ARKit depth estimation for close-range objects. Thus, scenarios with small objects require LiDAR or a custom neural network.
Solving the occlusion problem
When a 3D character walks through a real table, the user immediately loses trust. Correct occlusion requires depth ordering: for each pixel, know what is closer — the virtual object or the real surface. The solution: depth estimation (LiDAR or neural network) creates an occlusion mask. Pixels where real depth is less than virtual depth are rendered as real. This requires synchronization of depth and RGB streams with jitter below 5 ms. On devices without LiDAR we use monocular depth (MiDaS v3.1) plus semantics to refine object boundaries. Accuracy is lower, but acceptable for large objects. Compared to in-house development, our solution saves 30% on average.
How we build semantic scene understanding
Semantic segmentation AR classifies each pixel (floor, wall, chair, person). Models: SegFormer, Mask2Former trained on ADE20K or ScanNet. In production, we often fine-tune them for specific customer categories with 100–500 labeled frames.
Plane detection finds horizontal and vertical planes. ARKit / ARCore provide built-in detection, but for inclined surfaces or curved walls we use custom RANSAC algorithms.
Depth estimation AR from RGB. We use DPT, MiDaS, UniDepth. On devices with LiDAR we fuse RGB + LiDAR for ±1–2 cm accuracy.
Light estimation AR evaluates direction and intensity. ARKit provides spherical harmonics from HDR estimate. Neural approaches (EfficientLit, DiffusionLight) are more accurate in complex lighting situations.
Technical model details
- MiDaS v3.1: backbone EfficientNet-L, resolution 384x384, latency ~30ms on iPhone 14. - DPT: Vision Transformer, resolution 384x384, accuracy 15% higher but latency ~60ms. - SegFormer: MiT-B2, 20 classes, mIoU 0.45 on ADE20K.Scene understanding components
| Component | Base module (ARKit/ARCore) | Custom (our development) |
|---|---|---|
| Planes | Built-in, up to 30 planes | RANSAC + ML for arbitrary surfaces |
| Depth | LiDAR / ARCore Depth API | MiDaS/DPT + fusion for ±1 cm |
| Semantics | ARKit (6 classes) | SegFormer (20‑50 custom classes) |
| 6DoF detection | Not built-in | FoundationPose (with CAD) |
| Occlusion | Built-in depth-based | Semantically refined mask |
Ensuring real-time performance for scene understanding
Visual SLAM AR is the core of any AR: the system simultaneously builds a map and determines pose. Classic: ORB-SLAM3 (CPU-friendly). Neural SLAM: DROID-SLAM, Point-SLAM — more accurate on complex textures, but require GPU. For production on smartphones, we use ARKit / ARCore as the SLAM base and add custom CV models via Metal (iOS) or Vulkan (Android).
// ARKit: getting depth map and plane detection func session(_ session: ARSession, didUpdate frame: ARFrame) { // Depth estimation if let depthMap = frame.sceneDepth?.depthMap { // CVPixelBuffer with float32 depth values in meters processDepth(depthMap) } // Semantic segmentation (ARKit 4+) if let segBuffer = frame.segmentationBuffer { // Mask with classes: floor, wall, seat, window, door, table, face, person processSemantics(segBuffer) } } Platforms and tools
| Platform | SLAM | Depth | Semantics |
|---|---|---|---|
| iOS (ARKit) | Built-in | LiDAR / Neural | Built-in (limited) |
| Android (ARCore) | Built-in | Depth API | None (custom) |
| HoloLens 2 | Mixed Reality | Time-of-Flight | Scene Understanding API |
| Apple Vision Pro | visionOS | LiDAR + stereo | RoomPlan / MeshAnchor |
| Custom (Jetson) | ORB-SLAM3 | Stereo / ToF | Custom SegFormer |
More details in Apple ARKit Scene Understanding documentation.
What's included in developing a scene understanding module
- Analysis of AR scenarios and stack selection (ARKit, ARCore, custom)
- Development/adaptation of depth, semantics, detection models
- Occlusion integration with stream synchronization correction
- Performance tuning: target 30 FPS (latency p99 < 33 ms)
- API documentation and calibration instructions
- Training materials for the client's team
- 3 months of post-deployment support
Timeline and cost
Basic module (planes + depth + occlusion): 4–8 weeks, $15,000–$25,000. Full semantic scene understanding with custom categories: 10–16 weeks, $40,000–$80,000. Compared to in-house development, our solution saves 30% on average. For a detailed assessment of your scenario, contact our engineers.