AI car damage recognition by photo in mobile app

AI Car Damage Recognition from Photos in Mobile Applications

Insurance inspector photographs damaged car—AI instantly localizes scratches, dents, cracks. More than "finding and marking": for insurance acceptance, system must deliver reproducible, justified results resistant to fraud attempts.

Task: Detection, Segmentation, Classification

Three analysis levels:

Detection—bounding box around damage. YOLOv8 or RT-DETR work well if trained on appropriate dataset (CarDD, COCO-format markup with 6–8 classes: scratch, dent, crack, broken_glass, paint_damage, deformation, missing_part).

Segmentation—per-pixel damage mask. Instance segmentation gives area in pixels → with known scale → area in cm². YOLOv8-seg, Mask R-CNN.

Severity classification—surface scratch, deep scratch, dent, structural damage. Determines repair scenario (polishing, body repair, replacement).

// iOS: request damage analysis backend
struct DamageAnalysisRequest: Codable {
    let imageBase64: String
    let vehicleInfo: VehicleInfo?     // make, model, year—for context
    let captureMetadata: CaptureMetadata
}

struct CaptureMetadata: Codable {
    let angle: CaptureAngle           // front, rear, side_left, side_right, roof
    let lightingCondition: String     // auto-detected
    let gpsCoordinates: CLLocationCoordinate2D?
    let timestamp: Date
    let deviceModel: String
}

Capture metadata isn't optional detail for insurance. Geolocation and timestamp create digital trail hindering old damage claims as new.

Multi-Angle Capture Protocol

One photo insufficient for full damage assessment. Proper implementation—guided photo flow:

enum DamageInspectionStep: CaseIterable {
    case overview_front           // general front view
    case overview_rear            // general rear view
    case overview_side_left       // left side
    case overview_side_right      // right side
    case damage_closeup_1         // close-up #1 (user indicates area)
    case damage_closeup_2         // close-up #2
    case odometer                 // mileage
    case vin                      // VIN number

    var instruction: String { /* ... */ }
    var requiredDistance: DistanceRange { /* approx 2m, 0.3m, etc */ }
}

AR overlay shows where to stand and which area to capture—via ARKit/ARCore positioning. Reduces retake percentage from bad angles.

Fraud Detection

Insurance fraud is real. App-level checks:

struct AntifraudChecks {

    // 1. EXIF metadata: photo taken now, not from gallery
    func isLiveCapture(_ image: UIImage) -> Bool {
        guard let exifData = image.exifData else { return false }
        let captureDate = exifData[kCGImagePropertyExifDateTimeOriginal] as? String
        return isWithinLastMinutes(captureDate, minutes: 5)
    }

    // 2. GPS check: coordinates match claimed accident location
    func isLocationConsistent(_ metadata: CaptureMetadata, claimedLocation: CLLocation) -> Bool {
        guard let gps = metadata.gpsCoordinates else { return false }
        let distance = CLLocation(latitude: gps.latitude, longitude: gps.longitude)
                         .distance(from: claimedLocation)
        return distance < 500 // allow 500m
    }

    // 3. Photo of photo detection (photo of screen with someone else's damage)
    func isScreenPhoto(_ image: UIImage) -> Bool {
        // Analyze moiré patterns and screen pixel grid
        return moareDetector.detect(image) > 0.7
    }
}

Backend: Damage Detection

Model runs on GPU. For production load—TorchServe or Triton Inference Server.

# YOLOv8-seg inference for damage
from ultralytics import YOLO

model = YOLO("car_damage_seg_v8x.pt")  # x-variant for max accuracy

def analyze_damage(image_path: str) -> DamageReport:
    results = model.predict(
        image_path,
        conf=0.25,          # lower confidence threshold
        iou=0.45,           # NMS threshold
        imgsz=1280,         # high resolution critical for small scratches
        retina_masks=True   # high-accuracy masks
    )

    detections = []
    for result in results[0].boxes:
        mask = results[0].masks[i] if results[0].masks else None
        detections.append(DamageDetection(
            class_name=model.names[int(result.cls)],
            confidence=float(result.conf),
            bbox=result.xyxy[0].tolist(),
            mask_area_px=mask.area if mask else None,
            severity=classify_severity(result.cls, result.conf)
        ))

    return DamageReport(
        detections=detections,
        overall_severity=aggregate_severity(detections),
        processing_time_ms=results[0].speed["inference"]
    )

imgsz=1280 instead of default 640 is critical for small scratches (2–5mm in photo). At default resolution, surface scratches detect ~40% of time, at 1280—75%+.

Result Visualization

// Android: overlay damage annotations on photo
@Composable
fun DamageAnnotationView(
    image: ImageBitmap,
    detections: List<DamageDetection>
) {
    Box {
        Image(bitmap = image, contentDescription = null)
        Canvas(modifier = Modifier.matchParentSize()) {
            detections.forEach { detection ->
                // Bounding box with color by severity
                val color = when (detection.severity) {
                    Severity.MINOR -> Color(0xFF4CAF50)
                    Severity.MODERATE -> Color(0xFFFFC107)
                    Severity.MAJOR -> Color(0xFFFF5722)
                    Severity.STRUCTURAL -> Color(0xFFD32F2F)
                }
                drawRect(
                    color = color,
                    topLeft = detection.bbox.topLeft(size),
                    size = detection.bbox.size(size),
                    style = Stroke(width = 3f)
                )
                // Label with class and confidence
                drawDamageLabel(detection, color)
            }
        }
    }
}

Timeline Estimates

Backend with YOLOv8 detection and basic mobile client—2–3 weeks. Full system with guided photo flow, AR positioning, anti-fraud checks, segmentation, damage area estimation, insurer CRM integration, iOS + Android—1–3 months depending on integration requirements.