AI Defect Detection via Mobile App Camera

Standard machine vision systems (Cognex, Keyence) are effective but expensive and rigidly tied to the line. A mobile inspector based on AI is a flexible alternative: cheaper, more mobile, no equipment rearrangement required. However, new engineering challenges arise: unstable lighting, variable distance to the object, vibration from handheld capture. We solve them with on-device AI tailored to your process. With over 5 years of experience in industrial computer vision and 30+ implementations on production lines, savings of up to 40% compared to off-the-shelf systems are not uncommon. Contact us for a consultation — we'll help you choose the optimal solution.

AI Defect Detection via Mobile App Camera: Technical Challenges

Every millisecond counts, and internet in the workshop is a luxury. On-device inference eliminates latency and network dependency. Model size is limited by device RAM, speed by throughput requirements.

// iOS: industrial defect detection via CoreML
class DefectDetectionEngine {
    private let model: VNCoreMLModel
    private var confidenceThreshold: Float = 0.5 // adjustable on the test stand
    private var iouThreshold: Float = 0.45
    // Dedicated queue for stable FPS
    private let inferenceQueue = DispatchQueue(
        label: "defect.inference",
        qos: .userInteractive
    )
    func analyze(sampleBuffer: CMSampleBuffer) async throws -> [DefectDetection] {
        guard let pixelBuffer = CMSampleBufferGetImageBuffer(sampleBuffer) else {
            throw DefectError.invalidFrame
        }
        return try await withCheckedThrowingContinuation { continuation in
            inferenceQueue.async {
                let request = VNCoreMLRequest(model: self.model) { req, error in
                    if let error = error {
                        continuation.resume(throwing: error)
                        return
                    }
                    let detections = (req.results as? [VNRecognizedObjectObservation])?
                        .filter { $0.confidence >= self.confidenceThreshold }
                        .map { obs in
                            DefectDetection(
                                type: DefectType(rawValue: obs.labels.first?.identifier ?? "") ?? .unknown,
                                confidence: obs.confidence,
                                boundingBox: obs.boundingBox, // normalized [0,1]
                                severity: self.classifySeverity(obs)
                            )
                        } ?? []
                    continuation.resume(returning: detections)
                }
                request.imageCropAndScaleOption = .scaleFill
                let handler = VNImageRequestHandler(cvPixelBuffer: pixelBuffer)
                try? handler.perform([request])
            }
        }
    }
}

Performance. YOLOv8n in CoreML on iPhone 14: ~15 ms per inference (66 FPS potential). YOLOv8s: ~25 ms. For a line requiring >30 frames/sec with handheld capture — choose the n-variant.

Image Stabilization for Handheld Scanning — AI Defect Detection

Hand vibration is the enemy of small defects. Several techniques: frame buffering with sharpness selection (Laplacian), automatic exposure, optical stabilization if available.

// Frame buffering + selecting the sharpest frame
class StabilizedFrameSelector {
    private var frameBuffer: RingBuffer<CMSampleBuffer> = RingBuffer(capacity: 8)
    private var sharpnessScores: [Float] = []

    func addFrame(_ buffer: CMSampleBuffer) {
        let sharpness = computeLaplacianVariance(buffer)
        frameBuffer.push(buffer)
        sharpnessScores.append(sharpness)
    }

    // For analysis — take the frame with peak sharpness from the last N frames
    var bestFrame: CMSampleBuffer? {
        guard let maxIdx = sharpnessScores.indices.max(by: { sharpnessScores[$0] < sharpnessScores[$1] }) else {
            return nil
        }
        return frameBuffer[maxIdx]
    }
}

Also: AVCaptureDevice.activeVideoMinFrameDuration + exposureMode = .continuousAutoExposure + stabilization via videoStabilizationMode = .cinematic.

How to Fine-Tune the Model on Real Data?

Ready-made datasets for specific production are unavailable. You need your own labeling. The process:

1. Capture 200–500 samples on the production line (normal + defective)
2. Label in Label Studio or CVAT (bounding boxes + defect classes)
3. Augmentation: brightness ±30%, rotation ±15°, horizontal flip, Gaussian noise — simulating real shooting conditions
4. Train YOLOv8s/m (depending on speed requirements)
5. Convert to CoreML (.mlpackage) or TFLite
6. Iterative fine-tuning on production errors — every 2–4 weeks

# Fine-tuning on new production data
from ultralytics import YOLO

model = YOLO("defect_detection_v2.pt")  # previous version as base
results = model.train(
    data="production_defects.yaml",
    epochs=50,
    imgsz=640,
    batch=16,
    lr0=0.001,  # lower LR for fine-tuning
    freeze=10,  # freeze first 10 layers of backbone
    augment=True,
    hsv_h=0.015,
    hsv_s=0.7,
    degrees=10.0,
    translate=0.1,
    scale=0.5,
    mosaic=1.0
)

Every 2–4 weeks we fine-tune the model on new production data, increasing accuracy up to 98%. Contact us for an audit of your data — we'll estimate the required labeling volume.

How Do We Integrate the Solution with Your Systems?

The mobile inspector must record results in the MES/ERP system. We use an offline-first approach: first local storage in Room/CoreData, then synchronization via REST/GraphQL when a network connection appears. This guarantees no inspection is lost.

// Android: sending inspection result
data class InspectionResult(
    val productId: String,
    val batchId: String,
    val inspectorId: String,
    val timestamp: Instant,
    val detections: List<DefectDetection>,
    val verdict: InspectionVerdict, // PASS, FAIL, REVIEW
    val imageUrl: String, // saved photo with annotations
    val deviceId: String
)

suspend fun submitInspection(result: InspectionResult) {
    // First — local queue (production may be without Wi-Fi)
    localQueue.enqueue(result)
    // Sync when network appears
    syncManager.triggerSync()
}

Offline-first is critically important: workshop Wi-Fi is unstable, loss of an inspection result is unacceptable.

Challenge: Specifics of Industrial Quality Control

Defects in production vary drastically by industry. Below are examples of the most common cases.

For each industry — its own model. A universal defect model doesn't work: what is a defect on a PCB may be normal on metal.

What Does Our Work Include?

All components are configured for your production processes.

Timeline Estimates

MVP with a baseline model (200–300 labeled samples), on-device inference, local inspection history — 3–4 weeks. Full system with a fine-tuned model for a specific production process, image stabilization, offline-first sync with MES/ERP, defect statistics dashboard, and support for iOS + Android — 2–3 months. Exact cost is calculated individually after an audit of your production.

Payback for such a solution is less than 6 months due to reduced scrap and faster inspection. Contact us for an evaluation of your project. Order a pilot implementation today.

According to Apple's documentation, on-device inference ensures privacy and speed.