AI for Spatial Computing
Developing applications for Apple Vision Pro is a challenge for CV engineers: users control with gaze and gestures, and latency must be under 12 ms. Without an optimal AI architecture, the app becomes uncomfortable or non-functional. For example, running YOLOv8n on the Neural Engine can increase latency to 15 ms, causing user disorientation. We have implemented over 30 AI solutions for Spatial Computing on Vision Pro and Meta Quest — from visualization to industrial AR systems. Our team's experience spans all key platform versions, ensuring knowledge of the pitfalls.
How does hand tracking work?
Vision Pro features M2 and R1 chips. The R1 handles sensor streams: 12 cameras, LiDAR, 5 sensors. End-to-end latency from camera to render is under 12 ms — a hardware guarantee that cannot be replicated in software.
Key CV capabilities via visionOS API:
- Hand tracking — 27 joints per hand via ARKit 6 HandAnchor
- Eye tracking — gaze accuracy ~1°, for hands-free UI interaction
- Scene understanding — PlaneDetection, MeshAnchor, RoomPlan
- World anchoring — persistent placement of objects in space
Why Core ML is the only path for custom models
Core ML is the sole way to run custom ML models on-device. Conversion using coremltools:
import coremltools as ct import torch traced_model = torch.jit.trace(pytorch_model, example_input) mlmodel = ct.convert( traced_model, inputs=[ct.TensorType(name="input", shape=ct.Shape(shape=(1, 3, 224, 224)))], compute_precision=ct.precision.FLOAT16, compute_units=ct.ComputeUnit.ALL ) mlmodel.save("CustomModel.mlpackage") The Neural Engine on M2 delivers 15.8 TOPS — enough for classification (EfficientNet-B0: ~2 ms), detection (YOLOv8n: ~8 ms), and segmentation. Optimizing ComputeUnit configuration and quantization accelerates standard models by 30%. Core ML achieves 2× lower latency compared to ONNX on similar hardware.
How to convert a model to Core ML
Export your model from PyTorch to TorchScript via torch.jit.trace. Then use coremltools.convert with FLOAT16 precision and specify compute_units=ct.ComputeUnit.ALL. Save as .mlpackage and test on a real device.
Hand tracking comparison: Vision Pro vs Quest 3
Hand tracking on Vision Pro is twice as accurate as Quest 3 (27 joints vs 21). Eye tracking is unique to Apple. This provides more natural control and low latency. Investments in such a project typically pay back within 8–12 months through reduced operator errors. Budget is calculated individually based on task complexity.
Practical scenarios
Industrial AR — equipment inspection. The technician looks at a unit, the system recognizes it (object detection + ID matching), overlays an AR schematic with instructions directly onto the part. A custom model for part detection is trained on CAD synthetics and real photos.
Retail visualization — furniture fitting. RoomPlan builds a floor plan, PlaneDetection finds the floor, SLAM tracks position, and the 3D model is stably placed in world coordinates.
Medical training — anatomical 3D structures on a physical mannequin. Uses 6DoF registration via QR marker or pose estimation.
What is included in AI solution development?
- Prepared Core ML model (mlpackage) with FLOAT16 quantization
- Model card documentation: metrics, latency, limitations
- Integration code for visionOS (Swift/Objective-C)
- Training of the client's team on model usage
- Post-release support for 3 months
How AI solution development proceeds
- Pipeline architecture: model selection, data collection, metrics.
- Training: synthetic data (CAD, 3D) + real labeling, fine-tuning for the task.
- Integration via Core ML: conversion, quantization, testing on Neural Engine.
- Optimization: p99 latency, power consumption, preventing drops.
- Device testing: use cases, edge conditions.
- Documentation: model card, instructions, repository access.
- Post-release support: bug fixes, adaptation to new visionOS versions.
Platform comparison: Vision Pro vs Meta Quest 3
| Parameter | Apple Vision Pro | Meta Quest 3 |
|---|---|---|
| Chip | M2 + R1 | Snapdragon XR2 Gen 2 |
| Hand tracking points | 27 joints | 21 joints |
| Eye tracking | Built-in, gaze ~1° | Not present in base SDK |
| ML framework | Core ML (CPU/GPU/Neural Engine) | ONNX, TensorFlow Lite, Unity Sentis |
| Camera latency | <12 ms | ~20 ms |
Platform limitations
Core ML requires fixed input size — dynamic shapes are not supported. Custom Reality Compositor provides limited camera access. LiDAR is accurate up to 5 m (error 1-2%), beyond that error increases. Precision tasks (medical, industrial) require additional markers. Incorrect quantization increases error — validation on a test set is mandatory.
Development timelines
| App type | Timeline |
|---|---|
| 3D product visualization (retail) | 6–10 weeks |
| Industrial AR with custom detector | 10–18 weeks |
| Analytical platform with eye tracking | 8–14 weeks |
Cost is calculated individually — depends on CV task complexity and integration scope. Assess your project: contact us for a consultation. Get a detailed analysis of your task in 1–2 days. We guarantee involvement of senior engineers: 5+ years of experience in Spatial Computing, over 30 completed projects. Leave a request on our website — we will propose the optimal solution.