Implement On-Device Semantic Image Search with CLIP

Imagine a user types "dog on the beach" and the app instantly shows all photos of a dog on the beach—no tags, no manual sorting. That's semantic image search powered by CLIP (Contrastive Language-Image Pre-training) from OpenAI. We implement it entirely on-device: images and texts are converted into 512-dimensional vectors, and cosine similarity between them determines relevance. No data leaves the device—user privacy is preserved. Our experience with MobileCLIP-S0 and vector databases allows us to deploy such a solution in 1–4 weeks.

Evaluate the potential for your app. Savings compared to cloud-based search reach 80% due to zero inference costs. On-device CLIP search is 5x faster than cloud thanks to local inference. The ROI for on-device AI typically ranges from 6 to 12 months. Typical project cost: from $5,000 for basic integration to $15,000 for a full solution. Get a consultation to discuss details.

Architecture and Benefits of On-Device AI

The pipeline consists of two independent stages. Indexing: one-time for the whole gallery, then incremental. For each photo, compute a CLIP Image Embedding (512-dimensional vector) using a ViT-B/32 backbone, normalized to unit L2 norm, and store it in a local vector database. Search: on every user query, convert the user query into a CLIP Text Embedding (same 512-dimensional vector), perform ANN search (e.g., HNSW) for nearest vectors in the database, return photos sorted by descending cosine similarity.

On-device AI outperforms cloud: no network latency, zero inference cost, full user privacy. For an app with 10,000 daily active users, cloud costs can exceed $5,000 per month, while on-device costs nothing—up to 80% savings compared to cloud solutions, quick ROI.

Integrating CLIP via CoreML

Apple hasn't included CLIP in the standard Vision framework, but Apple ML Research released ml-mobileclip—a distilled version specifically for mobile devices. MobileCLIP-S0: 18 MB, 3–5 ms per image inference on iPhone 14. Source: Apple ML Research, MobileCLIP.

MobileCLIP on GitHub (official repository with code).

CLIP uses a contrastive learning objective to align image and text embeddings in a shared multimodal space, enabling zero-shot classification and retrieval. The model employs a contrastive loss that pulls matching image-text pairs together and pushes non-matching ones apart in the embedding space.

import CoreML class MobileCLIPEmbedder { private let imageEncoder: MobileCLIPImageEncoder private let textEncoder: MobileCLIPTextEncoder func embedImage(_ cgImage: CGImage) throws -> [Float] { let resized = resize(cgImage, to: CGSize(width: 256, height: 256)) let input = MobileCLIPImageInput(image: MLMultiArray(from: resized)) let output = try imageEncoder.prediction(input: input) return l2Normalize(output.embedding.toFloatArray()) } func embedText(_ query: String) throws -> [Float] { let tokens = tokenize(query) // BPE tokenizer let input = MobileCLIPTextInput(tokens: MLMultiArray(from: tokens)) let output = try textEncoder.prediction(input: input) return l2Normalize(output.embedding.toFloatArray()) } } 

The tokenizer for CLIP is BPE (Byte Pair Encoding). A Swift implementation is available in the ml-mobileclip repository. On Android: ONNX Runtime with MobileCLIP—less straightforward but works.

How to Integrate CLIP into an iOS App?

Step-by-step guide for implementing semantic search:

1. Prepare the model. Download MobileCLIP-S0 from the Apple ML repository. Convert to CoreML using coremltools.
2. Integrate the encoders. Create classes for processing images and text, as in the example above. Ensure normalization and tokenization match.
3. Set up indexing. Use BGProcessingTask for background indexing of the gallery. Save embeddings to a vector database (e.g., sqlite-vss).
4. Implement search. On receiving a query, compute the text embedding and perform ANN search. Return sorted results.

Example of background indexing:

class GalleryIndexer { private var lastIndexedDate: Date { get { UserDefaults.standard.object(forKey: "lastIndexedDate") as? Date ?? .distantPast } set { UserDefaults.standard.set(newValue, forKey: "lastIndexedDate") } } func indexNewPhotos() async { let fetchOptions = PHFetchOptions() fetchOptions.predicate = NSPredicate(format: "creationDate > %@", lastIndexedDate as CVarArg) let newPhotos = PHAsset.fetchAssets(with: .image, options: fetchOptions) newPhotos.enumerateObjects { [weak self] asset, _, _ in guard let self else { return } if let embedding = self.computeEmbedding(for: asset) { self.vectorDB.insert(assetId: asset.localIdentifier, embedding: embedding) } } lastIndexedDate = Date() } } 

Search completes in ~20 ms: text embedding (5 ms) + ANN search (15 ms). Results are instant.

func search(query: String, topK: Int = 30) async throws -> [PHAsset] { let textEmbedding = try mobileCLIP.embedText(query) let results = vectorDB.search(vector: textEmbedding, limit: topK) let fetchOptions = PHFetchOptions() fetchOptions.predicate = NSPredicate( format: "localIdentifier IN %@", results.map { $0.assetId } ) let assets = PHAsset.fetchAssets(with: fetchOptions) let idToScore = Dictionary(uniqueKeysWithValues: results.map { ($0.assetId, $0.score) }) return assets.objects(at: IndexSet(0..<assets.count)) .sorted { idToScore[$0.localIdentifier, default: 0] > idToScore[$1.localIdentifier, default: 0] } } 

How We Choose a Vector Database on Device

For searching among 50,000 vectors, an ANN index is needed. ANN indexes like sqlite-vss use quantization and hierarchical navigable small world (HNSW) graphs to accelerate search. Consider three options with concrete characteristics:

SQLite with the sqlite-vss extension adds virtual tables for vector search. Compact, works in embedded mode:

CREATE VIRTUAL TABLE photo_embeddings USING vss0(embedding(512)); INSERT INTO photo_embeddings(rowid, embedding) VALUES (42, json('[0.1, -0.3, ...]')); SELECT rowid, distance FROM photo_embeddings WHERE vss_search(embedding, json('[0.2, -0.1, ...]')) LIMIT 20; 

Simple flat L2/cosine via Accelerate for galleries up to 10k photos is sufficient without a specialized index:

func cosineSimilarity(_ a: [Float], _ b: [Float]) -> Float { var dotProduct: Float = 0 vDSP_dotpr(a, 1, b, 1, &dotProduct, vDSP_Length(a.count)) return dotProduct // After L2-normalization, equals cosine similarity } 

Brute-forcing 10,000 512-dimensional vectors on iPhone 14 via vDSP_dotpr takes ~15 ms. Acceptable for galleries up to 20k.

Multilingual Search

CLIP is trained predominantly on English. For a Russian query "собака на пляже", quality is worse than for "dog on beach". Solution: translate the query using a simple dictionary of common words or Google Translate API before embedding. In practice, translating 100–200 frequent queries offline is sufficient.

What's Included and How Long Does It Take?

Cost is calculated individually based on integration complexity and target devices. Get a consultation for an accurate estimate.

Why Trust Us with This Task?

Our team has 6+ years of experience in mobile ML and has delivered 15+ on-device AI projects, establishing a strong track record since 2018. We guarantee compliance with App Store Review Guidelines (sections 4.2 and 5.1) and user data security. We provide full documentation and post-deployment support.