Why visual search by photo does not find the product
The user points the camera at a product or uploads a screenshot — and instead of the right item gets empty results or a dozen cards from another category.
Technically, visual search did its job: the image was accepted, the vector representation was computed, and the response was returned in fractions of a second.
The failure is not in speed but in meaning — the system searches for a similar image, not the same product in your catalog.
The first cause is that the model computes the vector over a general image set where shoes, bags, and furniture are mixed together.
On your product cards it confidently finds “something shoe-like,” but does not distinguish model, color, and material, so catalog relevance drops at the very first step.
The second is the quality of the cards themselves. A single studio photo on a white background does not cover the angles in which a customer photographs the product: at an angle, under yellow light, with a shelf background. If the catalog has no shots from different sides, photo search simply has nothing to compare with.
The third is the index. While it is rebuilt manually once a week, new items do not appear in results at all. The share of empty results grows, the customer returns to text search or goes to a competitor — and this is already lost sales, not a technical trifle.
We break down how this is addressed: fine-tuning on your own cards, shooting requirements, and event-based rather than scheduled reindexing.
What changes in metrics after launching visual search
We calculate visual search metrics from a baseline taken on historical traffic: empty result rate, click-through to the product card, returns, support requests, and query response time.
Without a fixed reference point, any growth looks like search merit, so we take the measurement before launch and repeat it on the same query sample.
The empty result rate is the main indicator. Text search does not cover queries like “same but with a different print”: the user leaves without ever reaching the catalog.
The vector index on FAISS searches by photo embedding, not by word match, so results are filled even with wording that lacks exact terms, and the share of empty results drops severalfold, not by percentages.
Returns and support react next. When the buyer relies on the product photo rather than a text description, “not what I expected” happens less often, and the share of such returns decreases.
Typical requests like “help me find something similar” are intercepted by the results themselves — the load on first-line support drops.
We keep response speed predictable: catalog embeddings are computed in advance; when an item is updated, only the changed part is recomputed; and a ready index serves live queries.
Response time fits within fractions of a second on catalogs with hundreds of thousands of items — search does not become a bottleneck under peak load.
Comparison of selection modes: by photo, by link, from camera
The modes differ not by interface but by what enters the input: photo upload gives a ready frame, camera capture gives a stream of blurry and duplicate shots, and a link gives someone else’s card with watermarks and a cropped product.
The same vector index behaves differently under these conditions, so mode limitations are built into the architecture in advance rather than discovered on live results.
The common part is one: object detection and cropping, background normalization, vector extraction, and nearest neighbor search in the index followed by reranking.
The differences are in preprocessing and thresholds: for camera capture, sharpness estimation and frame deduplication are added; for a link, elements of someone else’s layout are cut out; for photo upload, there is a check that the object is recognized at all.
| Mode | When it is suitable | Limitations |
|---|---|---|
| Uploading a photo from the gallery | The product is fully in frame; image-based selection and similar products are needed | The result depends on angle, background, and cropping; without normalization the false match rate grows |
| Camera capture in the app | Product search in a store when you do not know the name and SKU | A stream of duplicates and blurry frames; best-frame selection and aiming prompts are needed |
| Selection by link | You do not have your own card, and the image is pulled from an external source | Someone else’s layout, watermarks, cropped product; the source may return the wrong frame |
A combination of modes is the norm: the camera is used for the “here and now” scenario, the gallery for repeat search, and links for products outside the catalog. We make the choice based on the share of such queries and on which attributes your card actually has.
Work order: catalog audit, index schema, quality check
The order of stages is built so that each next step relies on verified data, not assumptions. We start with a catalog audit — it determines both the quality of visual search and the amount of work on image preparation.
- Catalog audit. We export the cards, look for duplicates, products without photos, broken links, and inconsistent angles. The output is a category map with the real number of items suitable for training.
- Reference images. For each category, we select products with multiple angles and manually label them as reference. This is the base by which we then measure whether the system finds the right product from an amateur photo.
- Matching schema. We determine how a shot becomes a feature vector, which index we build, and which features we use to filter out obviously irrelevant items. Separately, we decide what to do with products that were not in training.
- Quality check. We run the system on a held-out part of the catalog: we calculate the hit rate in the top ten results, separately by category and for difficult cases — similar models, close colors, shooting in poor light.
- Catalog integration. We configure index updates when products appear and are removed, so that results do not become stale and do not require a full rebuild.
- Handover to support. We provide regulations, the index schema, and a report on result quality, and we train the customer’s team to reindex the catalog and handle complaints about search results.
Deliverables: indexes, schemas, access policies, regulations
Indexes, schemas, and regulations are the part of the delivery that lets the customer’s team develop search on its own, without us.
We hand over not only a working service but also a set of artifacts by which it can be reproduced, extended, and passed to a new engineer inside the company.
- Visual feature index: assembly of vector representations across all catalog cards with metadata and category labels, plus a description of full and incremental rebuild.
- Data schema: product — image — feature — category relationships, photo quality requirements, and rejection rules so that the index is not polluted with duplicates and stock images.
- Access policies: roles for the storefront, analytics, and admin, and separation of rights for reindexing, exporting vectors, and changing result thresholds.
- Update regulations: rebuild triggers when a product appears, is discontinued, or changes category, the procedure, and the window during which search remains on the old build.
- Quality evaluation loop: a set of reference queries “photo → expected results,” a top-hit metric, and a report on it for each new version.
- Rollback regulations: how many previous index and model builds are stored, how to return to the previous one, and in what time the results return to the prior state.
- Team instructions: launch, monitoring, analysis of discrepancies in results, and a checklist for accepting changes from content managers.
A separate document is the decision log: why this category labeling and this threshold for filtering similar products were chosen, what was tried and rejected. It saves months by not re-traversing already passed forks.
Case: the empty result rate in the catalog dropped severalfold
At a marketplace, the bottleneck was not in the model but in the features.
Photo search essentially amounted to category recognition and substitution of a text query: if the shot was not taken in studio conditions, the system returned an empty page or random items from the category.
The empty result rate held at 21%, top-3 hit rate was 46%, and after the first screen the user almost always went to regular text search.
What changed
We built a unified vector space for the shot and the product card: we took OpenAI CLIP as the basis, fine-tuned it on pairs “user photo — product from order” from logs, and added a second stage — reranking the top-50 candidates by attributes: color, material, silhouette.
The model outputs a vector, search goes by index, so the response fits within fractions of a second even on a catalog with hundreds of thousands of items.
What became
Metrics before and after were measured on the same query stream: empty result rate — 2.7%, top-3 accuracy — 88%, and the share of orders started from the photo scenario grew fourfold.
We catch regression on an offline set of 50 thousand pairs: if a new embedding version drops top-3 accuracy below the threshold, the rollout stops and automatically rolls back to the previous one.
How photo-based selection differs from attribute-based selection
Photo-based selection begins with the shot being turned into a vector representation, and the system searches for the nearest items in the catalog index. The buyer does not need to know the model name — a magazine shot, a screenshot from a chat, or a photo from a shelf is enough. The approach wins where a feature is hard to describe in words: silhouette, texture, print, shade.
Attribute filters work the opposite way — they narrow the choice by explicit, pre-labeled features: color, material, size, compatibility, characteristic range. The result is predictable and explainable: it is visible why an item got into the results.
But attributes must be filled and normalized, otherwise the filter silently cuts off half the catalog.
The boundary of applicability runs along the nature of the feature. If the decision relies on visual similarity — clothing, furniture, decor, parts by appearance — photo-based selection gives more relevant results than word search.
If numeric conditions decide, the photo is useless: a visually similar part may not fit by fit or connector.
In practice, the approaches work together on the same catalog. The photo forms a broad pool of candidates by similarity, and attributes narrow it to compatible options and allow sorting by availability.
The reverse route is also in demand: the buyer selects products with filters, opens a card, and asks to show similar ones.
Together they reduce the empty result rate and remove the dead end of “I do not know what this is called.”
Technical doubts: photo quality, cold start, load
Engineering doubts surface before launch, and almost all of them come down to data, not the model. Below are four questions that we resolve at the assessment stage: photo quality, cold start, load, and catalog update.
Half the catalog was shot on a phone under different light — will search make mistakes?
Preprocessing — cropping by object detection, white balance, uniform scale — removes part of the variance. Then similarity is calculated by semantic image representations (a CLIP-like model), so background and angle affect the results less. We fine-tune weak categories on your own “photo — product” pairs.
There are no labels or click history — where should cold start begin?
We start on an open pretrained model: “photo — description” pairs are collected from the cards, then we mix in clicks and search logs. The first working result appears on tens of thousands of items, not after collecting millions of labeled pairs.
Will search withstand a sale peak?
We keep the index in memory and shard it across replicas; repeated queries are served by cache. At peak, a two-stage scheme turns on: fast approximate retrieval, then exact recomputation of the top results — response time stays even, rather than growing linearly with traffic.
How do we update the catalog without rebuilding the index entirely?
Incrementally: a new card enters search minutes after publication, and a discontinued item leaves by the lifecycle rule. A full rebuild runs on a separate loop on schedule — the old and new index versions work in parallel, and rollback is done by switching.
Let’s discuss the task: send the catalog and query examples
To return with an estimate and plan rather than general considerations, we need data about the catalog and about how customers search. The more specific the inputs, the more accurately we name timelines, stages, and acceptance metrics.
- Catalog export in CSV or JSON: SKU, name, category, attributes, photo links — it shows what share of cards are suitable for indexing and which ones will need cleaning.
- 50-100 example queries, including typos, slang, and Latin characters — we use them to build result quality checks.
- Examples of photo queries: the product in full, on a shelf, in an interior, in low light — search robustness depends on this.
- Reference labeling for 100-200 cards: which products are considered the correct answer. Without it, accuracy cannot be measured.
- Current search metrics: share of queries without a result, clicks to the product card, orders from search — the baseline for comparison.
- Infrastructure constraints: where the catalog lives, how often stock changes, and whether automatic photo export is available.
- The responsible person on your side: who confirms the labeling and accepts the work stages.
After that, the work order is transparent: we fix scenarios and metrics, build a prototype on your catalog, compare it with the reference, then integrate and roll out to production. Describe the task and send the listed items — we will return with an estimate and plan.
