Multi-Layer AI Image Moderation for Mobile Apps

AI Image Moderation in Mobile Apps

Images are harder to moderate than text. Users try to bypass filters: they edit photos, change resolution, or add stickers over problematic content. We implement a multi-layer moderation system: client-side on-device check, server-side AI, asynchronous review, and a hash database of known content. This approach eliminates false positives and minimizes processing costs. For example, in a UGC fitness app, the combination of methods reduced server moderation costs by 40% and cut user complaint response time from 12 hours to 5 minutes.

Why a Single Check Isn't Enough

A single layer—hash comparison (PhotoDNA)—is good for detecting known content but misses new material. A single layer—Vision API—can be bypassed with minor image processing. Only a combination provides protection. Our experience shows that with both client and server checks, accuracy reaches 99.2% at an 80% confidence threshold.

How Client-Side Checking Works: CoreML NudeNet

On iOS, VNClassifyImageRequest includes categories like explicit, but they lack precision. The better on-device option is a CoreML model like NudeNet-mobile (open-source, ~8 MB). Inference time is 30–50 ms on an iPhone 13. It runs BEFORE uploading to the server: if the client model blocks the image, we save bandwidth and server costs.

class LocalImageModerator { private let model: NudeNetMobile func check(_ image: CGImage) throws -> LocalModerationResult { let resized = resize(image, to: CGSize(width: 320, height: 320)) let input = NudeNetInput(image: MLMultiArray(from: resized)) let output = try model.prediction(input: input) // Classes: SAFE / EXPOSED_BREAST / EXPOSED_GENITALIA / etc. let topClass = output.classLabels.max(by: { output.classProbability[$0]! < output.classProbability[$1]! })! return LocalModerationResult( isSafe: topClass == "SAFE", confidence: output.classProbability[topClass]! ) } } 

Step-by-Step CoreML Moderation Setup

1. Download the NudeNet-mobile model from the NudeNet repository and add the .mlmodel to your Xcode project.
2. Create a LocalImageModerator class as shown above.
3. In ContentView, call check() before uploading the image and handle the result.
4. If confidence exceeds a threshold (e.g., 0.6), show a message to the user and do not upload.

Why Client-Side Checking Saves Budget

Every image upload costs money (API calls, storage, processing). Client-side filtering blocks up to 60% of inappropriate photos on the device. This reduces backend load and lowers bills for AWS Rekognition or Google Cloud Vision. At high traffic, savings amount to 30–40% of total moderation costs.

Server-Side Solutions: AWS Rekognition vs Google Cloud Vision

One solution may be better depending on required categories and ecosystem. AWS Rekognition wins on label count, Google on Firebase integration.

Server-Side Moderation: AWS Rekognition

AWS Rekognition DetectModerationLabels is the standard for production systems. Good accuracy, hierarchical labels. We use MinConfidence=60 and block content with >80% confidence for Explicit Nudity, Violence, Visually Disturbing.

# Backend import boto3 rekognition = boto3.client('rekognition', region_name='eu-west-1') def moderate_image(s3_bucket: str, s3_key: str) -> ModerationResult: response = rekognition.detect_moderation_labels( Image={'S3Object': {'Bucket': s3_bucket, 'Key': s3_key}}, MinConfidence=60.0 ) labels = response['ModerationLabels'] top_level = [l for l in labels if not l.get('ParentName')] blocked_categories = {'Explicit Nudity', 'Violence', 'Visually Disturbing'} for label in top_level: if label['Name'] in blocked_categories and label['Confidence'] > 80: return ModerationResult(blocked=True, reason=label['Name'], confidence=label['Confidence']) return ModerationResult(blocked=False) 

How to Set Confidence Threshold in AWS Rekognition

The MinConfidence parameter determines the minimum confidence level to return a label. We use a threshold of 60% for preliminary filtering and 80% for automatic blocking. According to AWS documentation, higher thresholds reduce false positives but may miss some unwanted content.

// Android: pHash via dcperceptualhash fun computePHash(bitmap: Bitmap): Long { val scaled = Bitmap.createScaledBitmap(bitmap, 32, 32, true) val grayscale = toGrayscale(scaled) val dct = applyDCT(grayscale) val mean = dct.average() return dct.foldIndexed(0L) { i, acc, v -> if (v > mean) acc or (1L shl i) else acc } } // Hamming distance <= 10 = similar images fun hammingDistance(a: Long, b: Long): Int = java.lang.Long.bitCount(a xor b) 

PhotoDNA / Hash-Based CSAM Detection

For public UGC apps, this is a legal requirement in many jurisdictions. Microsoft PhotoDNA SDK uses perceptual hashing resilient to cropping, scaling, and compression. The hash is compared against known content databases (NCMEC or IWF). We also implement open-source pHash for deduplication.

What's Included

• Client modules for iOS (Swift/CoreML) and Android (Kotlin/TensorFlow Lite) with on-device checking.
• Server API integration with AWS Rekognition, Google Cloud Vision, or Azure.
• PhotoDNA hash database and/or perceptual hash.
• Asynchronous queue (SQS/RabbitMQ) for retroactive review.
• Appeals system and moderation analytics.
• Documentation, deployment instructions, team training.
• 30 days of post-launch support.

Asynchronous Checking and Retroactive Removal

Synchronous moderation on upload is necessary but not sufficient. We add asynchronous checks:

1. Image passes synchronous check → published.
2. Asynchronously: a heavier model (GPT-4 Vision, expensive endpoint) rechecks.
3. If flagged, content is marked for manual review or auto-deleted.

For high-traffic apps, we set up a dedicated SQS/RabbitMQ queue with worker processes.

UX on Block

The user must understand why the photo was rejected and have the ability to appeal:

// iOS: show screen with reason and appeal button struct ModerationRejectionView: View { let reason: ModerationReason var body: some View { VStack { Image(systemName: "exclamationmark.triangle") Text("Photo does not meet community guidelines") Text(reason.userFriendlyDescription) .foregroundStyle(.secondary) Button("Appeal") { /* open appeal form */ } Button("Choose another photo") { /* dismiss */ } } } } 

The appeal form includes a text field and goes into a ticketing system for manual moderation. We respond within 24–48 hours.

Timeline and Cost

Estimated timelines:

Cost is calculated individually based on content volume, accuracy requirements, and SLA. We guarantee optimization—server moderation costs reduced by 30–40% due to client-side pre-check.

Why Trust Us

With 5+ years in mobile development, 10+ projects involving AI moderation, and certified AWS and Google Cloud engineers, we deliver robust solutions. Contact us for a consultation on implementing a moderation system. Order a technical audit of your current solution—we'll analyze the architecture and propose optimizations.