AI exercise repetition counting via mobile camera

AI-Powered Exercise Rep Counting via Mobile Camera

Rep counting sounds simple but breaks on edge cases. User squats slowly—one rep. Pulses at top—model counts two. Phone tilted—coordinates shifted, counter lost.

Foundation: Pose Estimation + Time Series Analysis

Same approach as posture analysis: VNDetectHumanBodyPoseRequest (iOS) or ML Kit Pose Detection (Android) output skeleton key points each frame. But for rep counting, we care about movement of key point over time, not static pose.

For each exercise, select tracking point:

class RepCounter {
    private var positionHistory: [Double] = []
    private var repCount: Int = 0
    private var state: MovementState = .neutral
    private let minAmplitude: Double = 0.08  // 8% screen height

    enum MovementState { case neutral, goingDown, bottom, goingUp, top }

    func update(normalizedY: Double) {
        positionHistory.append(normalizedY)
        if positionHistory.count > 30 { positionHistory.removeFirst() }

        let smoothed = positionHistory.suffix(5).reduce(0, +) / 5
        detectRep(smoothedY: smoothed)
    }

    private func detectRep(smoothedY: Double) {
        let baseline = positionHistory.prefix(10).reduce(0, +) / 10
        let deviation = smoothedY - baseline

        switch state {
        case .neutral where deviation > minAmplitude:
            state = .goingDown
        case .goingDown where deviation < minAmplitude * 0.3:
            state = .bottom
        case .bottom where deviation > minAmplitude * 0.7:
            state = .goingUp
        case .goingUp where deviation < 0:
            state = .top
            repCount += 1
            onRepCompleted?(repCount)
            state = .neutral
        default: break
        }
    }
}

5-frame rolling average (suffix(5))—smooths pose estimation jitter. Without it, counter twitches from point tremor between frames.

Calibration Per Exercise

minAmplitude = 0.08—percent of screen height. Works for squats, needs more for biceps curl (wider arm range), less for push-ups and different axis.

Calibrate via analytics (train on labeled rep videos) or adaptive baseline: first 3 seconds of exercise—measure movement amplitude, adjust thresholds.

class AdaptiveRepCounter {
    private var calibrationPhase = true
    private var calibrationSamples: [Double] = []
    private var dynamicAmplitude: Double = 0.05

    func calibrate(y: Double) {
        calibrationSamples.append(y)
        if calibrationSamples.count >= 75 {  // ~3 sec at 25fps
            let range = calibrationSamples.max()! - calibrationSamples.min()!
            dynamicAmplitude = range * 0.4  // 40% observed amplitude
            calibrationPhase = false
        }
    }
}

Camera Orientation: Front vs Side View

Same task requires different solutions by camera position:

Front view (camera before user): visible shoulders, hips, head. Good for squats, lunges, burpees. Hip Y-coordinate well correlates with squat phase.

Side view: knee, hip, ankle visible in profile. Better for analyzing knee bend angle (squat form), but requires extra stand / tripod—inconvenient in practice.

Most apps optimize for front view with "place phone 2 meters away at waist level" instruction.

Problem: Similar Exercises

Model doesn't know which exercise user performs. If user selected "squat" but does lunge—counter calculates wrong. Solutions:

1. User explicitly selects exercise—simple, works
2. Auto-detect exercise—separate classification task

For auto-detection use time-series classifier: LSTM or 1D CNN on joint angle sequences. Datasets: NTU RGB+D (120 action classes), UCF101. Convert to CoreML/TFLite for on-device classification.

Feedback and UI

Rep counter—central element. Large, contrasting, animated. Per counted rep: UIImpactFeedbackGenerator(style: .rigid) + visual counter flash.

Overlay over camera: display skeleton (lines between key points). Helps user understand model "sees" and counts correctly. Red skeleton on poor visibility (low point confidence).

Development Process

Integrate pose estimation per platform. Implement rep detection algorithm for target exercise set. Adaptive calibration. Camera UI with overlay skeleton. Test on real people of different builds, different phones.

Timeframe Estimates

Count 3–5 basic exercises—1–2 weeks. Auto-detect exercises + complete workout tracking—3–5 weeks.