AR scene reconstruction 3D mesh building

Implementing 3D Scene Reconstruction (Scene Reconstruction Mesh) in AR

Scene Reconstruction is not just "AR sees floor". It's a live environment mesh that ARKit updates in real-time as camera moves. ARMeshAnchor accumulates room geometry, surfaces get classified, app gets data to work with: detect obstacles, build navigation graphs, run physical simulation.

Implementing this without killing performance is separate task.

Mesh Architecture and Main Pitfalls

ARMeshGeometry stores vertices, normals and triangle indices. Updates via session(_:didUpdate:) delegate — each frame ARKit can send dozens of updated ARMeshAnchor. Naive implementation recreating MeshResource or SCNGeometry on each update kills main thread in seconds.

Correct approach: update mesh only for changed anchors, use MDLMesh as intermediate format and pass data to Metal buffers directly. In RealityKit looks like:

func session(_ session: ARSession, didUpdate anchors: [ARAnchor]) {
    for anchor in anchors.compactMap({ $0 as? ARMeshAnchor }) {
        updateMeshVisualization(for: anchor)
    }
}

func updateMeshVisualization(for anchor: ARMeshAnchor) {
    let geometry = anchor.geometry
    // Work with geometry.vertices, geometry.faces directly
    // Don't create new MeshResource every time — patch existing
}

Second pitfall — surface classification. ARMeshClassification gives: .floor, .ceiling, .wall, .door, .window, .seat, .table, .none. But classification only works with sceneReconstruction = .meshWithClassification and only on LiDAR devices. Without checking ARWorldTrackingConfiguration.supportsSceneReconstruction(.meshWithClassification) — crash or silent ignoring.

Third — mesh coordinates in world space. ARMeshGeometry.vertices give local coordinates relative to ARMeshAnchor.transform. To get world coordinates multiply each vertex by anchor's transformation matrix. Forgot — mesh renders in wrong place.

How We Build Scene Reconstruction

Basic stack: ARKit 5+ + RealityKit 2 + Metal. Don't use SceneKit for mesh — not optimized for dynamic geometry.

Session setup:

let config = ARWorldTrackingConfiguration()
config.sceneReconstruction = .meshWithClassification
arView.debugOptions = [.showSceneUnderstanding] // for debugging
arView.session.run(config)

For mesh visualization in debug mode draw wireframe via arView.debugOptions. In production hide mesh visibility, but use data for:

• Occlusion — objects behind walls invisible
• Physics — CollisionComponent interacts with real geometry
• Raycast — accurate hit on real surfaces, not just planes

Case from practice: warehouse navigation AR app. Needed to detect obstacles (shelves, pallets) and build route. Used Scene Reconstruction for occupancy grid: each mesh vertex with .none classification (unrecognized object) added to obstacle graph. NavMesh updated every 2 seconds — balance between freshness and CPU load. On iPad Pro M2 keeps 60 FPS without drops.

Fallback and Diagnostics

On non-LiDAR devices Scene Reconstruction unavailable. Offer degradation to plane detection with manual mesh capture via ARPlaneAnchor. Worse, but better than blank screen.

For mesh quality diagnostics — ARView.debugOptions.insert(.showSceneUnderstanding): green wireframe shows what ARKit sees. Useful testing non-standard environments (glossy floors, mirrors — LiDAR works poorly due to reflection).

Timeline

Basic integration with mesh visualization — 1–2 weeks. If need surface classification, physical collisions and mesh-based navigation — 4–6 weeks. Cost calculated after detailed discussion of requirements.