AI Object Tracking for iOS and Android: Development and Optimization

Imagine: your mobile app counts visitors in a store, but every time a person steps out of the frame and returns, the system counts them as new. This is the classic error of naive detection without tracking — loss of object identity. Take a real case: automating visitor counting in a shopping mall using entrance cameras. Without tracking, the system erroneously counted the same person multiple times if they briefly left the field of view. After implementing ByteTrack with Kalman filter, false positives dropped by 90%. We solve this by integrating AI tracking that links objects across frames and preserves their unique IDs. Besides ID switches, issues arise with occlusions (object temporarily hidden) or fast motion (object shifts beyond the frame edge). From the start, it's important to understand: tracking is not just detection on each frame, but association of objects over time. According to ByteTrack, using low-confidence detections reduces ID switches by 30-40%.

Why tracking is harder than detection?

A detector determines an object's class and position on each frame independently. A tracker answers the question: 'Is this the same object as in the previous frame?' Main challenges:

• Occlusions: object temporarily hidden by another object or obstacle.
• Crossings: two objects swap positions — the tracker may swap IDs.
• Fast motion: object shifts beyond the frame by more than its bounding box size.

We use two approaches: SOT (single object) and MOT (multiple objects). The choice depends on the scenario.

What are SOT and MOT?

SOT (Single Object Tracking)

The user taps an object — the app follows it. Applications: sports broadcasts, tracking a specific person in the frame, AR games. Algorithms: SiamFC, OSTrack, STARK. On iOS — Vision VNTrackObjectRequest.

MOT (Multi-Object Tracking)

Simultaneous tracking of all objects of a required class. Applications: visitor counting, traffic monitoring, production conveyors. Algorithms: SORT, ByteTrack, StrongSORT, OC-SORT.

Why ByteTrack is more reliable than SORT?

SORT only uses detections with confidence above a threshold. ByteTrack uses all detections, even low-confidence ones. This sharply reduces track loss:

// Android: ByteTrack association class ByteTracker( private val trackThresh: Float = 0.5f, private val highThresh: Float = 0.6f, private val matchThresh: Float = 0.8f ) { private val trackedStracks = mutableListOf<STrack>() private val lostStracks = mutableListOf<STrack>() fun update(detections: List<Detection>): List<STrack> { val highDetections = detections.filter { it.confidence >= highThresh } val lowDetections = detections.filter { it.confidence in trackThresh..<highThresh } val (matches1, unmatched_tracks1, unmatched_dets1) = linearAssignment(trackedStracks, highDetections, matchThresh) val (matches2, _, _) = linearAssignment(unmatched_tracks1, lowDetections, 0.5f) val newTracks = unmatched_dets1.map { STrack(it) } return (matches1 + matches2).map { it.track } + newTracks } } 

ByteTrack reduces ID switches by 30-40% compared to SORT during frequent occlusions. Meanwhile, computational complexity remains low — the tracker runs on CPU without noticeable heating.

How does the detector-tracker pipeline work on iOS?

Standard pipeline for mobile:

// iOS: YOLOv8 detection + SORT tracking class MultiObjectTracker { private let detector: YOLOv8Detector private let tracker: SORTTracker // SORT parameters — important to tune for the task init(targetClass: String, maxAge: Int = 10, // frames without detection before deleting track minHits: Int = 3, // frames of detection to confirm track iouThreshold: Float = 0.3) { self.detector = YOLOv8Detector(targetClass: targetClass) self.tracker = SORTTracker(maxAge: maxAge, minHits: minHits, iouThreshold: iouThreshold) } func processFrame(_ pixelBuffer: CVPixelBuffer) async -> [TrackedObject] { let detections = await detector.detect(pixelBuffer) let tracks = tracker.update(detections: detections.map { det in Detection(bbox: det.boundingBox, confidence: det.confidence) }) return tracks.map { track in TrackedObject( id: track.trackId, boundingBox: track.bbox, isConfirmed: track.hitStreak >= tracker.minHits, velocity: track.kalmanFilter.velocity ) } } } 

maxAge = 10 — the track lives 10 frames without detection (object behind obstacle). At 30 FPS, that's 333 ms — enough for brief occlusions.

How to implement ByteTrack in 5 steps?

1. Choose detection model: YOLOv8-nano (INT8) for mobile — 2x faster, mAP drops by 1-2%.
2. Tune tracker: set trackThresh=0.5, highThresh=0.6, matchThresh=0.8.
3. Integrate pipeline: detection on each frame, tracking after filtering low-confidence detections.
4. Render tracks: via Metal/OpenGL — up to 60 FPS on mid-range devices.
5. Optimize: lower FPS to 15-20 if accuracy is not critical — saves 40% energy.

Common mistakes and how to avoid them

// iOS: ByteTrack pipeline (simplified) class ByteTrackPipeline { private let detector: YOLOv8Detector = .init() private var tracker: ByteTracker = .init() func process(pixelBuffer: CVPixelBuffer) async -> [Track] { let detections = await detector.detect(pixelBuffer) let tracks = tracker.update(detections: detections) return tracks } } 

How to reduce CPU load during AI tracking?

Use quantized models (INT8) for detection — speedup up to 2x without noticeable mAP drop. Trackers SORT and ByteTrack are lightweight by themselves, runnable on CPU. Render bounding boxes via Metal (iOS) or OpenGL (Android) to offload the main thread. On iOS, connect Core ML with Neural Engine; on Android, NNAPI. If accuracy is not critical, reduce video stream FPS to 15-20 — saves up to 40% energy.

What's included in our tracking work

• Task analysis: choosing approach (SOT/MOT), target classes, occlusion scenarios.
• Prototyping: training or fine-tuning detection model, selecting tracker.
• Module development: integrating detector and tracker, track rendering, camera orientation handling.
• Optimization: low power consumption, working at 30 FPS on mid-range devices.
• Testing: 50+ scenarios (lighting changes, fast motion, crossings).
• Support: documentation, source code, team training.

We have 5+ years of mobile development experience and have implemented tracking for retail, logistics, and sports. Get an engineer consultation — we'll assess your project and propose a solution.

Timeline estimates

Contact us for a consultation and precise timeline estimate. We guarantee results and provide post-delivery support.