AR game mechanics implementation in mobile app

Implementing AR Game Mechanics in Mobile Applications

AR games aren't just 3D objects on screen. This is physics, interaction with real world, managing scene lifecycle and working when user moves, covers camera with hand, ARKit loses tracking at worst time.

Any mechanic looking simple in 2D or 3D games, in AR requires rethinking.

Physics in AR: RealityKit vs SceneKit vs Custom

RealityKit — first choice for new projects. PhysicsBodyComponent + CollisionComponent give native physics via ARKit's scene understanding:

var physicsBody = PhysicsBodyComponent(massProperties: .default, material: nil, mode: .dynamic)
physicsBody.isAffectedByGravity = true
entity.components[PhysicsBodyComponent.self] = physicsBody

var collision = CollisionComponent(shapes: [.generateBox(size: entity.visualBounds(relativeTo: nil).extents)])
entity.components[CollisionComponent.self] = collision

With enabled arView.environment.sceneUnderstanding.options = [.physics] AR objects correctly interact with real surfaces. Ball rolls on real table, box stands on real shelf.

RealityKit problem — limited control. No access to individual physics engine steps, no custom force generators. For simple mechanics — sufficient. For complex game physics (ragdoll, fluid simulation, rope physics) — need SceneKit + PhysicsWorld or Unity with AR Foundation.

Control in AR: Gestures vs Controller

Tap to place, pinch to scale, rotation — standard. But in AR gestures compete with camera movement. User wants to rotate object — ARKit interprets as hand tracking.

Solution via UIGestureRecognizer with shouldRequireFailure(of:): tap recognized only if swipe doesn't start. For rotation — UIPanGestureRecognizer with minimum displacement 20 points before activation, so accidental camera movement doesn't trigger rotation.

Separate mechanic — "aiming" via screen center instead tap. More convenient in shooters and strategies: raycast from arView.center each frame, object under crosshair highlights, action — button. Implemented via ARView.raycast(from:allowing:alignment:) in CADisplayLink callback.

Most Frequent AR Game Mechanic Problems

Tracking loss during gameplay. ARKit transitions to .limited tracking state — objects "float". For AR games this disaster: enemy teleports 30 cm, ball passes through wall. Strategy: on .limited "freeze" physics, show overlay "Point camera at surface", resume after .normal. Don't interrupt game loop — just pause physics.

AR objects outside viewport. Player turns, AR enemy behind back continues moving and attacking. Need frustum culling not for rendering (ARKit does this), but game logic: AI of enemies outside viewport can work simplified or pause.

Spawning on uneven surfaces. Raycast returns surface normal via ARRaycastResult.worldTransform — fourth column is point, X/Y/Z — orientation. If surface tilted 30°, spawned object tilts with it. For games with "flat" objects (tokens, chips) fix vertical: take only Y from normal, build transform with up = (0,1,0).

Case from Practice

Mobile AR strategy: towers, enemies, real table as map. 6 unit types, pathfinding on mesh surface. Main problem: ARMeshAnchor updates asynchronously — while mesh rebuilds, pathfinding graph stale. Solution: navigation graph updated every 3 seconds on background thread via GCD, units used cached graph. Unit during movement checked collision with current mesh via raycast — caught situations "mesh changed, unit walks through wall".

FPS kept at 60 on iPhone 13. On iPhone 11 dropped to 45 with 6+ units. Solution: LOD for distant units (simplified meshes), shadow disable beyond 0.5 meters.

What's Included

• AR object physics with real surface interaction (RealityKit / SceneKit)
• Control system: gestures, aiming, UI buttons
• Game loop with ARKit tracking loss handling
• AI and pathfinding on AR surfaces
• LOD optimization for supporting iPhone 11 and up
• Testing in real lighting conditions

Timeline

Cost calculated after detailed mechanics discussion.