Picture this: a user snaps a photo of a jacket on the street, opens your app, and gets an exact or similar model from the catalog in 200 milliseconds. That's the task of visual fashion search, which we solve for fashion retailers. ASOS, Zara, and Pinterest use similar approaches, but custom catalogs require fine-tuning. Technically, visual search consists of two stages: offline indexing of all items into a vector space, and online nearest neighbor search. In practice, challenges arise: noisy user photos, the need to account for attributes (color, texture, cut), high load. We've accumulated experience on projects with catalogs up to 2 million SKUs. Order a pilot in 6–10 weeks and get a working solution — we guarantee quality thanks to certified engineers with 10+ years of experience.
Architecture of Visual Search for Clothing and Accessories
The task splits into two stages: offline — catalog indexing, building vector embeddings for all items using the CLIP model; online — extracting the embedding from a query image and finding nearest neighbors in an ANN index.
Embedding Models for Fashion
General image embedding models (CLIP, DINO) provide a decent baseline, but for visual fashion search, specialized models are better:
- Fashion CLIP — fine-tuned on 800K fashion pairs. Understands both text and visual queries.
- SES (Style Embedding Space) — contrastive learning with triplet loss.
- Metric learning: ArcFace or Proxy-NCA on pairs of SKUs of the same style.
| Model | R@1 (DeepFashion) | R@10 | Embedding Size |
|---|---|---|---|
| ResNet-50 (ImageNet) | 0.21 | 0.48 | 2048 |
| CLIP ViT-B/32 | 0.34 | 0.61 | 512 |
| Fashion CLIP | 0.47 | 0.74 | 512 |
| Fine-tuned ArcFace | 0.63 | 0.85 | 256 |
Based on DeepFashion benchmark (Liu et al., 2016)
How Clothing Segmentation Improves Search Accuracy?
Catalog product photos have a white background and studio lighting. User query photos show people in crowds with complex backgrounds. Extracting an embedding from the whole image causes the model to match on background. Solution: segment the clothing before extracting the embedding. Models: HumanParsing (SCHP, lip-parser), Grounded-SAM. After segmentation, crop the clothing with a neutral background. In an internal test with 50K queries, R@1 increased from 0.43 to 0.58.
Re-ranking via Attribute Matching
After ANN search, we perform attribute-based re-ranking on the top-100 results: extract attributes of the query image (color, pattern, silhouette, material) using a multi-label classifier trained on DeepFashion-C. Rank the top-100 by a combination of visual similarity and attribute match. This boosts P@5 by 12–15% and reduces returns by 15–20%, providing cost savings for your business.
Why Fine-Tuning on Catalog Data Matters?
Base models provide general understanding, but for your specific assortment, fine-tuning is needed. For example, if the catalog has unique prints, without fine-tuning R@1 can be below 0.3. We use ArcFace loss on pairs of similar/dissimilar items. For a catalog of 100K+ SKU, 10K labeled pairs suffice. Get a consultation on model selection.
Vector Index for Production
Catalog of 2M SKU = 2M vectors dim=256. Exact search is slow. We use approximate nearest neighbor (ANN):
- FAISS — IndexIVFPQ with 1024 clusters gives recall 0.92 at latency < 10 ms for 2M vectors on a single A10G.
- Qdrant or Weaviate — managed vector DB with metadata filtering.
- pgvector — if you already have PostgreSQL and catalog < 500K items.
| Index | Latency (p99) | Recall@10 | Scalability |
|---|---|---|---|
| FAISS IVFPQ | < 10 ms | 0.92 | 10M+ vectors |
| Qdrant | < 20 ms | 0.95 | 100M+ vectors |
| pgvector | < 50 ms | 0.98 | < 1M vectors |
Index Update Details
When adding a new SKU — inference of embedding model + upsert. Incremental updates.Multimodal Search: Text + Image
CLIP-based architecture allows searching by image + textual refinement: "similar dress, but red." It's implemented by combining embeddings. Contact us for a demo.
Work Process and What's Included
- Analytics: audit of catalog, accuracy and latency requirements.
- Design: model selection, index architecture.
- Implementation: fine-tuning, building ANN index, API.
- Testing: load testing, A/B test.
- Deployment and support: rollout, documentation.
Includes: audit, fine-tuning, segmentation, index, attribute re-ranking, integration, load testing, team training. We guarantee quality: certified engineers with 10+ years of experience, 30+ projects for e-commerce.
Timelines
MVP for catalog up to 100K SKU: 6–10 weeks. Production system for 1M+ SKU: 3–5 months. Contact us to assess your project — get a consultation on architecture and timelines.