Building Fitness Apps & Games with AR Body Tracking

Imagine you are building an AR fitness trainer. The user performs an exercise, and a virtual character must mirror the movement. If tracking is inaccurate, animation desyncs and the user loses motivation. Our engineers faced this while developing a squash app: a 3-frame delay made the game unplayable. The solution was to optimize the tracking pipeline for the specific device. We rewrote the handler using Metal, and the delay dropped to 1 frame.

We use ARKit on iOS (A12+) and MediaPipe on Android. Each platform has its own characteristics: ARKit outputs 91 skeleton joints with centimeter-level accuracy, while MediaPipe gives 33 key points with 5–10 cm error. The choice depends on your target audience and budget. Selecting the right platform can save up to 30% of the prototyping budget.

In this article, we will cover: how to obtain real-time pose, bind a 3D model to the skeleton, analyze joint angles for fitness, and compare platforms. All code snippets are ready for integration. We work with ARKit 6, RealityKit, MediaPipe 2.0, and Flutter. For clients requiring maximum accuracy, we recommend iOS with LiDAR — it adds depth information.

Technology Comparison

ARKit vs MediaPipe

iOS offers ARBodyTrackingConfiguration (requires A12+). The skeleton has 91 joints in a hierarchical structure. Code to get the wrist position:

func session(_ session: ARSession, didUpdate anchors: [ARAnchor]) { guard let bodyAnchor = anchors.first as? ARBodyAnchor else { return } let skeleton = bodyAnchor.skeleton if let wristTransform = skeleton.modelTransform(for: .rightHand) { let worldTransform = bodyAnchor.transform * wristTransform // Attach object to wrist } } 

Limitations: distance 1.5–5 m; single person in frame; fast movements cause 2–4 frame delay. For multi-person we use third-party ML solutions.

For Android we use MediaPipe Pose Landmarker (33 key points). Accuracy is lower than ARKit (5–10 cm vs 1–3 cm), but sufficient for fitness analytics. Human pose estimation AR is essential for accurate fitness tracking. Comparison:

ARKit is 3 times more accurate than MediaPipe in joint positioning. ARKit outperforms MediaPipe by 3x in joint positioning error. ARKit uses the Neural Engine and LiDAR data (on Pro models) for stable depth. MediaPipe relies solely on RGB images, so accuracy drops with complex backgrounds or poor lighting. We use inverse kinematics to refine joint positions. Transformations are computed using quaternions for precision.

Implementation

Attaching a 3D Character

Follow these steps:

1. Create a USDZ scene with a rig compatible with the ARKit joint hierarchy.
2. Use RealityKit's BodyTrackedEntity to apply skeleton transformations.
3. Verify joint names match using Reality Composer Pro.

If joint names don't match, the character "explodes." We verify this using Reality Composer Pro. Skeletal animation AR brings characters to life.

Movement Analysis for Fitness

Joint angle via dot product:

func jointAngle(joint1: simd_float3, vertex: simd_float3, joint2: simd_float3) -> Float { let v1 = normalize(joint1 - vertex) let v2 = normalize(joint2 - vertex) return acos(dot(v1, v2)) * (180 / .pi) } 

Knee angle when squatting: 80–110° is normal. Below 60° is too deep. We implement real-time feedback with voice prompts.

Advanced Features

Multi-Person Tracking

Standard ARKit does not support tracking multiple people. For that, we use MediaPipe Pose with a separate detector per person, optimized with GPU. On iPhone 15 Pro we achieve 30 fps for two people. On Android with NNAPI, we get 20–25 fps for two people. BlazePose is a real-time pose estimation model.

Performance Optimization

• Lower frame rate to 30 fps if accuracy allows.
• Use LOD for 3D models: reduce polygon count when far.
• Cache detection results when pose is static.
• On Android, use NNAPI or the MediaPipe GPU delegate.
• On iOS, use Metal for GPU computations — it reduces CPU load. On iOS, we leverage GPU compute shaders via Metal.

Services and Pricing

What's Included

• SDK with documentation and code examples for iOS/Android.
• Ready prototype with basic tracking and 3D object attachment.
• Integration of fitness analytics (angles, reps, calories burned).
• Rigging and animation of a 3D character to the skeleton.
• Performance optimization for specific devices.
• Post-release support: 1 month free.

Timelines and Cost

Basic body tracking with 3D object attachment: 1–2 weeks. Character rigging + integration: 3–4 weeks. Fitness analytics: 4–6 weeks. Android version (MediaPipe): additional 2–3 weeks. Development cost ranges from $5,000 to $20,000 depending on features. Book a consultation — we will find the optimal solution for your budget.

Why Choose Us

• 10+ years of experience in mobile development
• 50+ successful AR projects
• Guaranteed deadlines and NDA
• Certified specialists in ARKit and MediaPipe

Need precise body tracking for your project? Contact us for a free assessment — we will propose the best solution. AR body tracking on iOS and Android employs different techniques.