AI 3D object reconstruction via NeRF from photos in mobile app

AI-Powered 3D Reconstruction from Photos (NeRF) in Mobile Apps

NeRF (Neural Radiance Field) on a mobile device is not "press button, get 3D model." It is a multi-step process with serious computational demands that, as of 2024–2025, has finally become realistic for production applications through 3D Gaussian Splatting and new models like InstantNGP and Nerfacto.

NeRF vs Gaussian Splatting vs Photogrammetry

Three technologies solve the same problem: 3D objects from photos. The differences are fundamental:

For mobile applications in 2025: 3D Gaussian Splatting offers the best balance of speed and quality. For quick AR previews, photogrammetry with the latest COLMAP pipeline is recommended.

Architecture: Capture on Device, Process in Cloud

On-device reconstruction is only practical in limited scenarios (Apple Object Capture API—Mac with Apple Silicon only). The practical architecture for mobile:

Mobile Device:
  - Guided capture flow (20–60 photos on orbital path)
  - ARKit metadata (camera poses)—simplifies COLMAP SfM
  - Upload to cloud

Backend (GPU instance):
  - COLMAP SfM (if no ARKit poses) or pose estimation from metadata
  - 3D Gaussian Splatting training (nerfstudio + gsplat)
  - Export to .splat / .ply / .glb
  - CDN → mobile device

Mobile Device:
  - Download and render result
  - AR viewing via RealityKit (iOS) / SceneView (Android)

Guided Capture on iOS with ARKit

Key UX requirement: the user must orbit the object correctly, otherwise reconstruction will contain artifacts.

class GuidedCaptureSession: NSObject {

    private var arSession: ARSession
    private var capturedFrames: [(UIImage, simd_float4x4)] = []  // image + camera transform
    private let targetFrameCount = 40
    private let minAngleBetweenFrames: Float = 8.0  // degrees

    func shouldCaptureFrame(currentTransform: simd_float4x4) -> Bool {
        guard let lastTransform = capturedFrames.last?.1 else { return true }

        // Angular distance from last captured frame
        let angularDistance = computeAngularDistance(currentTransform, lastTransform)
        return angularDistance >= minAngleBetweenFrames
    }

    var captureProgress: Float {
        // Estimate orbit coverage around object
        let coveredAngles = estimateOrbitCoverage(capturedFrames.map { $0.1 })
        return min(coveredAngles / 360.0, 1.0)
    }
}

AR overlay displays the "orbit" around the object: green arcs show captured angles, gray areas show what still needs to be captured. This reduces failed reconstructions from incomplete coverage.

Image Quality Requirements

Before uploading to the cloud—basic validation:

func validateCaptureSet(_ frames: [(UIImage, simd_float4x4)]) -> ValidationResult {
    // Minimum frame count
    guard frames.count >= 20 else {
        return .insufficientFrames(current: frames.count, required: 20)
    }

    // Angular coverage (at least 270° out of 360°)
    let orbitCoverage = estimateOrbitCoverage(frames.map { $0.1 })
    guard orbitCoverage >= 0.75 else {
        return .insufficientCoverage(coverage: orbitCoverage)
    }

    // Average frame sharpness
    let avgSharpness = frames.map { sharpnessScore($0.0) }.reduce(0, +) / Float(frames.count)
    guard avgSharpness >= 60.0 else {
        return .blurryImages
    }

    return .valid
}

Backend: 3D Gaussian Splatting Training

# nerfstudio + gsplat pipeline
from nerfstudio.cameras.cameras import CameraType
from nerfstudio.pipelines.base_pipeline import Pipeline

def run_gaussian_splatting(
    images_dir: Path,
    camera_poses: list[np.ndarray] | None = None,
    output_dir: Path = Path("output")
) -> Path:
    """
    If camera_poses are provided (from ARKit)—skip COLMAP SfM.
    This reduces processing time from 15-20 min to 3-5 min.
    """

    config = SplatfactoModelConfig(
        num_downscales=2,       # reduce for speed
        use_scale_regularization=True,
        max_gauss_ratio=10.0,
    )

    trainer = Trainer(config, output_dir=output_dir)
    trainer.train()  # ~10-40 min on GPU (A100: 10 min, T4: 25 min)

    # Export to web-friendly formats
    export_gaussian_splat(output_dir / "splat.ply")
    export_glb(output_dir / "model.glb")  # for AR Quick Look / SceneViewer

    return output_dir

Displaying Results in AR

// iOS: RealityKit Quick Look for .usdz / .glb
import RealityKit
import ARKit

class ModelViewerViewController: UIViewController {

    func presentARModel(modelURL: URL) {
        let arView = ARView(frame: view.bounds, cameraMode: .ar)

        let anchor = AnchorEntity(plane: .horizontal)
        ModelEntity.loadModelAsync(contentsOf: modelURL)
            .sink(
                receiveCompletion: { _ in },
                receiveValue: { [weak self] entity in
                    entity.generateCollisionShapes(recursive: true)
                    anchor.addChild(entity)
                    arView.scene.anchors.append(anchor)
                    // Pinch to scale, pan to move
                    arView.installGestures([.scale, .translation, .rotation], for: entity)
                }
            )
            .store(in: &cancellables)
    }
}

Timeline Estimates

An MVP with guided capture, cloud upload, Nerfacto/Gaussian Splatting training, and basic AR viewing takes 3–4 weeks. A complete system using ARKit camera poses (no COLMAP), processing progress tracking, multi-format export (.glb, .usdz, .obj), 3D model sharing, and iOS + Android support requires 2–3 months.