Visual clothing search: where search results break
Visual clothing search breaks not on the model, but on the data: the catalog returns irrelevant product cards by photo, because matching runs on text attributes — category, color, and a short description from the product card. The user finds a similar item by eye, while the catalog answers with the warehouse's vocabulary.
Product matching by photo requires the image to become the primary key.
We describe the photo and the product card with a set of numerical features — cut, silhouette, fabric texture, print, hardware, fit — and image search runs on proximity in this space, not on substring matches in the title.
Typical errors in matching analogs are predictable. A phone shot in a fitting room has a different white balance than a studio photo on a mannequin, and the model starts confusing beige with gray and denim with wool.
Similar categories — an oversized shirt, a tunic, a long blouse — differ insignificantly in features and collapse into a single result set.
Catalog result quality is determined not by the algorithm's "intelligence" but by annotation and the cutoff threshold.
If the catalog has verified "photo — reference" pairs, you can measure the share of queries where the desired item appears in the first five product cards and raise it through iterations rather than rebuilding the entire system.
Below, we break down how such image search is structured: from catalog preparation and reference pairs to filtering out noise and quality control on live traffic.
Visual photo matching: measurable effects
Visual search stops being a showcase feature when metrics are captured separately from the overall catalog. Below are the effects we record on annotated streams: comparison is against text search and manual selection.
Share of queries closed on first display
A single model computes vector representations of photos and product cards, so matching runs on silhouette, cut, and texture rather than tags.
The share of sessions where an item goes to the cart from the first result set grows — this is also visible in overall catalog conversion, while the number of refining actions falls from four or five to one or two.
We work on a CLIP-like architecture: we fine-tune the model on the client's catalog, without collecting a dataset from scratch.
Empty results and matching speed
Previously, the share of queries with no result remained high: a text description poorly conveys print, cut, or shade. The visual index always responds — with a weak match, it returns the closest analogs by shape and color.
The share of empty result sets falls to single-digit percentages, and product matching speed drops to a fraction of a second per query.
Returns and annotation load
Returns due to size and style hit margin harder than other reasons, and visual matching addresses them directly: the buyer sees how the item fits on a similar figure and less often orders several positions "at random." At the same time, the load on manual annotation is removed — attributes are assigned automatically, and new positions enter search on the day of publication.
Recognition approaches: comparison of options
Photo-based matching is solved by three fundamentally different schemes, and the choice between them is determined not by model complexity but by how similar the catalog is to the data on which the model was trained.
An off-the-shelf model maintains acceptable accuracy on mass categories — basic knitwear, jeans, sneakers. It confuses custom cuts, rare prints, and handmade items.
Fine-tuning on the catalog removes most misses, but it requires annotated "photo — product" pairs and retraining when the collection changes significantly.
The hybrid scheme works in reverse: the photo is decomposed into features — color, category, cut, material, length — and matching runs on them and the text fields of the card.
The result set is easier to explain and correct with rules, but the scheme does not distinguish pattern and texture.
| Option | When it fits | Limitations |
|---|---|---|
| Off-the-shelf model without fine-tuning | Mass categories, fast launch | Does not know your categories, errors on rare items |
| Fine-tuning on the client's catalog | Custom assortment, stream of new photos | Requires annotation and retraining when the collection changes |
| Hybrid scheme: features and card text | Need explainability and manual selection rules | Does not distinguish pattern, texture, close shades |
| Similar-item search within the catalog | "Find the same, but cheaper" scenario | Does not find items that are not yet in the catalog |
At the start, we run the catalog through all available schemes and compare metrics: the share of queries where the desired item appears in the first ten positions, the share of empty result sets, and the share of manual edits. The decision is made based on these numbers, not on the name of the approach.
Stages of implementing an image-based matching system
Visual search breaks not on the model, but on the catalog: duplicate cards, different angles of the same item under different lighting, attributes without a single filling rule.
That is why the stages of implementing the system go from data to model — first a catalog audit, then search setup.
- Catalog audit. We count volume and structure: the share of duplicates and near-duplicates, which categories are shot from several angles, where the background interferes with matching. The output is a list of items that need to be re-shot or excluded from the index.
- Solution scheme and relevance metrics. We fix what the search should find: the same models, similar styles, an "item + accessory" set. The share of hits in the top result set and accuracy by category are agreed before setup — otherwise there is nothing against which to accept the result.
- Training sample preparation. We collect "image → correct product" pairs, clean duplicates, and mark difficult cases: solid-color clothing, a back view. We split the data into training, validation, and a holdout set that the model will not see until acceptance.
- Search setup. We configure feature extraction and the index structure, and select cutoff thresholds: when to show close matches and when it is more honest to return an empty result set.
- Validation on a holdout sample. We run scenarios, calculate relevance metrics, and analyze errors by category. If an entire group of items fails, we return to annotation rather than "tweaking" thresholds.
- Acceptance and handover. We provide documentation, access, a reindexing procedure for catalog replenishment, and a metrics report. We teach the client's team to run updates and read the error slice independently.
What we hand over after implementation: delivery scope
Handing over the solution is not an archive with source code, but a set of artifacts by which your team can maintain the system without us.
We record the delivery scope in the acceptance act and verify it at acceptance: the client's engineer independently deploys the stand according to our instructions and repeats typical operations.
- Service configurations — values of search and ranking parameters, load limits, cache settings, with an explanation of what changes as the catalog grows.
- Interaction schemes — visual search request flows, integration points with the catalog, image storage, and accounting systems.
- Access policies and roles — a permission matrix for the administrator, annotation operator, and analyst, rules for issuing and rotating keys.
- Index update procedure — rebuild frequency, the order of loading new items, and checking result completeness before publication.
- System documentation — module descriptions, deployment and rollback instructions, a catalog of typical failures with the order of actions.
- Set of reference queries — images with expected result sets, used to track search quality degradation after releases.
- Recovery plan — backup scheme, time, and order of returning to the previous working state.
We close each item with the question "can your team do this on its own." If the answer is "no," we add to the instruction rather than keeping the knowledge with us.
Case: how search relevance grew
The task came from clothing and accessories retailer Modario: a catalog of several tens of thousands of SKUs, photo search worked confidently only on studio shots, while on user-generated ones — different background, lighting, crop, angle — the result set fell apart.
Diagnostics showed three causes.
The index was built by a single model for the entire catalog without a category filter, so a dress searched for bags with a similar texture; embeddings were computed from the original frame without cutting out the object, and the background weighed more than the item itself; the cutoff threshold was set so that the system preferred to show at least something instead of an honest "not found."
What we changed: object cropping during query processing, a category classifier as a mandatory filter before ranking, index rebuilding on vector representations recalculated from annotated real queries, and a new cutoff threshold.
First we rolled it out to a share of traffic alongside the old result set and compared on live queries — rollback behind a flag remained one command away.
Result: the share of relevant impressions in the top 10 grew from 61% to 89%, the share of empty result sets fell from 8.4% to 1.2%, product matching time decreased from three minutes to forty seconds, and support requests "didn't find by photo" became five times fewer.
A relevance drop is now caught by daily monitoring: if the share of empty result sets rises above the threshold, traffic automatically returns to the previous index.
How does photo-based matching differ from tag-based matching?
Tag-based matching works with a discrete description: attribute annotation reduces an item to a set of labels from a finite vocabulary — category, color, length, sleeve type, fit.
The vocabulary inevitably coarsens: a single label "midi straight dress" covers both dense knitwear and a flowing viscose cut, although for the buyer these are different things. If the query matches the label, the item is found; if it does not match, the position is invisible.
The limitation here is not in the annotators' accuracy, but in the nature of the label. At the boundaries, two specialists disagree: one assigns "oversized," the other "loose fit," and the item stops being found by both queries.
Each new description axis expands the vocabulary and the amount of work, but texture, the character of the print, and the fit on the figure do not fit into it.
Photo-based matching compares not words, but image representations. The model builds a vector in which silhouette, cut, texture, and pattern are encoded simultaneously — no vocabulary is needed for this.
Similarity is calculated as the distance between vectors, and the vector index returns the closest positions in fractions of a second on a catalog of tens of thousands of items. Robustness to background and lighting comes from training on pairs of photos of the same item.
The difference is noticeable on queries that are not described by a vocabulary: a street photo, a screenshot from social media, "find the same, but without a belt." We measure matching accuracy on a holdout sample — the share of relevant items in the first ten and the share of queries where an analog from an adjacent category was found.
We do not discard tags: they support hard filters by size, category, and availability. The photo is responsible for "similar," and tags narrow it down to what can be ordered.
Technical doubts and engineers' answers
Before starting, clients check not the search itself, but how it will survive a real catalog: half-empty cards, visually similar items, seasonal waves, a change of photographer. The doubts are technical, and there is an engineering answer to each.
Half the items have no attributes — will search work?
Yes. Matching runs on the image, so cards without descriptions and categories enter the result set from day one. Attributes refine filters, and we fill them in after launch without stopping search.
Will similar models be confused with each other?
The system returns a ranked list rather than a single answer, so close silhouettes are an expected result. Annotating "similar / different" pairs helps separate them: using it, we configure which features to rely on — cut, length, texture.
What about new collections that were not in the index?
The index updates as the catalog is uploaded, so new positions appear in the result set after cards are published — no retraining is required. We cover seasonal waves with an update procedure: by the sales start date, the item is already searchable.
The catalog was re-shot in a different style — will quality drop?
We check on a control sample: change the background, lighting, and pose on old cards, and see whether the item holds its place in the result set. If it drops, we re-index the shoot — matching recovers without rebuilding.
How can matching quality be controlled after handover?
We leave a set of control queries with a reference result set and run them on a schedule. The report shows the share of hits in the top — degradation is visible before customer complaints, not after.
Let's discuss the task and come back with a plan
To come back with an estimate and work plan rather than general considerations, we need access to your data and an understanding of the scenarios.
Visual search is built around the catalog and how the buyer uses it: the same model will give different quality on a catalog of 500 positions with consistent shoots and on 200 thousand cards with different lighting and backgrounds. Below is what we request in the first conversation.
- Catalog export. A dump in your preferred format or read-only access to the database: SKU, category and subcategory, color, material, season, image link, size availability. This is enough to assess the suitability of the photos and the scope of work for normalizing them.
- Examples of unsuccessful matches. 15–30 live queries or photos where search, in your view, misses. They become a test set: it shows whether accuracy grows after each iteration, rather than "it seems better."
- Collection volume and update rate. How many positions there are now, how many are added per season, how often photos change and models go out of sale. This determines the reindexing strategy and how the system behaves on new items.
- Usage scenarios. Where exactly the buyer will encounter search: similar-item matching in a product card, photo search from a smartphone, matching a set to a selected item, a showcase for the seller. Each scenario has its own result logic and its own accuracy bar.
- Integration points. Website, mobile app, catalog, search, recommendation block, manager panel — with a flow diagram and an owner on your side. Here we also fix the test environment and load constraints.
- Success criterion. We agree in advance on what we consider a successful match: hitting the category, the color, the specific item. One metric at the start saves arguments at acceptance.
Describe the task and send access to the export — we will come back with an estimate and a stage-by-stage work plan.
