Product Deduplication When Importing from Multiple Suppliers

Why importing from multiple suppliers breaks the catalog

Each supplier delivers a price list in its own format: one has the SKU in a SKU column, another uses Code, and a third doesn't have it at all, with the product identified by name.

Importing price lists from multiple suppliers without normalization leads to the same product being created anew on every update: the case changes, the brand gets added to the name, the unit of measure differs.

This is how duplicates appear in the catalog, and from there they spread through the entire chain — into filters, search, and warehouse stock.

The cost of the error isn't visible right away.

A customer sees three cards for the same product with different prices and can't tell which one is real; a manager can't reconcile stock because part of it sits on one card and part on another; sales analytics gets smeared, and purchasing decisions are made on distorted data. Catalog integrity rests not on the operator's care but on matching rules that run on every import.

Product deduplication is not a one-time cleanup but a layer of logic between the price list and the catalog.

It determines which attributes make a product count as the same one (supplier SKU, barcode, brand + model combination), what to do when prices conflict, and how to merge descriptions without losing specifications.

Below we break down how this works and why, without such a layer, the catalog degrades with every new supplier.

What correct product deduplication delivers

Correct product deduplication is not a cosmetic catalog cleanup but a change to its physics.

When the same product arrives from three or four suppliers under different names, SKUs, and units of measure, the catalog starts growing faster than the assortment: for a hundred real positions, three hundred cards easily accumulate.

After merging by key attributes — manufacturer, model, modification, barcode — the number of cards shrinks severalfold, and this is the first thing the customer sees in the report.

The second effect is search conversion. While duplicates live as separate cards, stock is spread across them: one position shows zero, another shows three units, the buyer sees "out of stock" and leaves.

After merging, stock adds up into a single card, and the share of queries that end in an add-to-cart grows without a single change in advertising or prices.

The third effect is manual edits. Product deduplication removes the main source of constant work for a content manager: reconciling duplicates, transferring descriptions, manually recalculating stock after every delivery.

The merge rule is applied automatically on import, so new arrivals from suppliers land directly in existing cards instead of creating new ones.

Separately, there is the stock error — the most expensive class of failures. When two cards reference the same physical product, writing off one does not reduce the number on the other, and the buyer places an order for a position that is already gone.

Merging cards to a single identifier closes this gap: stock becomes shared, and the discrepancy between warehouse and storefront stops accumulating.

All three effects are measurable and can be verified against suppliers' historical data before implementation — we break down how this works and which merge rules your catalog specifically needs.

Comparison of product matching approaches

Product matching is a choice between completeness and precision: the more aggressively an algorithm merges records, the higher the risk of collapsing two different products into one.

In practice, the choice comes down to four approaches, and almost always the working configuration is a hybrid one: first exact matching by SKU, then fuzzy matching by name, while ambiguous cases go to manual review.

Exact matching yields a zero rate of false merges, but it covers only the part of the catalog where suppliers have agreed on identifiers.

Fuzzy matching raises coverage, but it requires a similarity threshold, normalization of case, units of measurement, and SKU prefixes — otherwise "cable 2 m" and "cable 2 m shielded" will stick together.

The hybrid approach with merge rules resolves this contradiction: exact matching covers the core of the catalog, fuzzy matching picks up the rest, and the rules determine which record the name, description, price, and stock are taken from.

Ambiguous pairs are not deleted silently but go into a manual review queue with a preserved history of decisions — this way an error is fixed once and does not return on the next import.

Stages of implementing deduplication into an existing import

Implementation happens on top of a working import, without stopping the intake of price lists. The first step is an audit: we look at which suppliers provide data and in what format, where duplicates have already accumulated, and by which fields products actually match.

This gives a real picture, rather than the assumption that "the SKU will solve everything."

1. Audit of current price lists. We collect exports from all suppliers, calculate the volume, the share of overlaps, and typical discrepancies — different brand spellings, extra spaces, SKUs with supplier prefixes.
2. Matching key schema. We determine what counts as a single product: the "brand + SKU" combination as the primary key, with the normalized name and barcode as secondary. We fix the priority of sources so that when fields conflict, it's clear whose value takes precedence.
3. Merge rules. We configure how cards are combined: which fields we take from the reference supplier, what we supplement, and how we handle price, stock, and specifications. The rules are described explicitly, not left "to the algorithm's discretion."
4. Testing on historical data. We run the new rules against accumulated price lists and see how many false merges and missed duplicates resulted. The match threshold is adjusted to an acceptable level, and disputed pairs are reviewed manually.
5. Implementation into the import pipeline. We connect deduplication to the current flow: new items are checked before being written to the catalog, while existing ones are updated rather than created anew.
6. Regulations and handover. We provide a description of the keys, merge rules, and the procedure for handling disputed matches — so the team can maintain the process without us.

After launch, the catalog stops growing due to duplicates, and price and stock updates run on a single card instead of several. Describe how your import is set up now — we'll come back with an estimate and a work plan.

What the client receives after setup

The deliverable is not a "configured module" but a complete package your engineers can work with without us: changing rules, adding suppliers, resolving ambiguous matches.

• Deduplication rule configurations: source priority, similarity thresholds for SKU, name, and attributes, behavior on field conflicts — which supplier is considered the master source.
• Mapping schemas: a map of each supplier's fields to your product model, including units of measure, nested categories, and composite attributes.
• Access policies: which employees can edit rules, who can only view the queue of ambiguous matches, who confirms manual merging of cards.
• Manual review queue: an interface for cases where automation is uncertain, with a decision history — so that a repeat import doesn't reopen an already closed issue.
• Documentation: a description of the matching logic, a glossary of supplier synonyms and abbreviations, and instructions for adding a new source.
• Update policies: how often thresholds are reviewed, who is responsible for cleaning up the reference catalog, and what to do in case of mass discrepancies after a supplier changes their export format.
• A set of checks on a test sample: reference "duplicate / not duplicate" pairs that show whether the rules have degraded after edits.
• Metrics: the share of automatically merged cards, the volume of the manual queue, the number of false merges — these show when it's time to review the rules.

We hand all of this over with a walkthrough on your data, not as an abstract template. Describe your task — we'll come back with an estimate and a plan.

Case study: how we eliminated duplicates in a catalog of 40,000 items

What it was before

The catalog of 40,000 items was assembled from the price lists of five suppliers, and each import added new cards instead of updating existing ones. The same product arrived under different names, SKUs, and units of measurement — suppliers did not cross-check with each other.

The catalog grew faster than sales: by the time of the review, it contained about 63,000 cards, of which nearly a third were duplicates.

Manual edits consumed the content manager's time: after each import, matches had to be reconciled by hand, and some duplicates still remained in the search results. The buyer saw three cards for the same product with different prices and could not tell which one was real.

What we changed

We built a matching pipeline on the application side: normalization of names and SKUs, comparison across a set of attributes — brand, model, unit of measurement, barcode.

Each pair is assigned a match weight, and based on a threshold the system decides whether to update an existing card or create a new one. Ambiguous cases are not dumped into a common pile but go into a separate queue for confirmation.

The import was rewritten to use batch processing with an idempotent key: re-running a price list no longer spawns duplicates but updates the same items. The logic is captured in rules, not in the manager's head — a new supplier can be connected without rewriting code.

What it became

Duplicates in the catalog stopped accumulating: the catalog now reflects the actual product range rather than the history of uploads. Manual edits were reduced to processing the confirmation queue — that is dozens of items instead of thousands.

The project was led by TrueTech: our engineers were responsible for both the matching logic and the restructuring of the import, so the deduplication rules immediately formed the foundation rather than being bolted on top of a finished catalog.

How deduplication differs from data normalization

Data normalization brings values to a unified form, but it does not decide what counts as a single product.

These are two consecutive processing stages, and confusing them means getting a catalog where "iPhone 15 128Gb" and "Apple iPhone 15 128 GB" remain two different listings.

Normalization answers the question "how should a value be written," while deduplication answers "how many records should remain."

Normalization works at the level of an individual field: it standardizes case and keyboard layout, removes extra spaces and non-printable characters, brings units of measurement to a single scale, splits merged attributes like "128Gb Black" into storage capacity and color, and cleans SKUs of supplier prefixes.

At the output, each card remains itself, but its fields become comparable — without this, the next step is impossible, because comparing "128 GB" and "128gb" directly is meaningless.

Data deduplication solves a different task: determining which records describe the same physical product and consolidating them into a single card.

Here a composite key is used — the normalized manufacturer SKU, brand, and model plus a set of characteristics — rather than a single field.

Two suppliers may name a listing differently, provide different photos and different package contents, but behind them is one product, and in the catalog it must be one.

The boundary between the approaches shows up in disputed cases. Normalization will not help if one supplier writes the SKU with a warehouse suffix and another without it: the values will remain different, even though the product is the same.

Conversely, deduplication without normalization will merge too much — "iPhone 15" and "iPhone 15 Pro" will match by brand and model if the characteristics are not compared in a unified form.

That is why the order is strict: first data normalization for each field, then deduplication by composite key, and only after that — merging cards with priority rules by supplier.

If the stages are swapped, some duplicates will go unnoticed, and some different products will collapse into one — and both classes of errors will have to be sorted out manually.

Common technical concerns

Import already works, products are in the catalog, and the thought "won't deduplication break what's already running" is a normal engineer's reaction. Below is a breakdown of the three concerns that most often come up before starting.

Suppliers have different units of measure — how do we reconcile them?

We convert to a base storage unit through a coefficient reference: pack, weight, volume, piece. The coefficient is set once per unit and reused — when adding a supplier, there's no need to rewrite the rules.

If a unit is ambiguous (for example, a "box" without nesting), the product goes into a manual review queue rather than being matched at random.

How do we avoid losing product history when merging duplicates?

History is a separate entity; it is not deleted along with the duplicate card. Prices, stock, and movements are tied to the matching group, not to a specific supplier record, so after a merge the price chart and sales statistics remain continuous.

Old identifiers are preserved as aliases — external systems and links continue to find the product by its previous code.

What if the algorithm makes a mistake and merges different products?

A merge is reversible: before applying, a snapshot of the groups is saved, and the process itself runs in batches with a decision log. Disputed pairs are not merged automatically — they go into a report with match indicators, and a human makes the decision. Rolling back a batch returns the cards to their original state without losing supplier data.

Describe how your import is currently set up and how many suppliers are in the environment — we'll get back to you with an estimate and a phased implementation plan.

Let's discuss your deduplication task

To come back with an estimate and a plan rather than generalities, we need access to the real picture of your catalog.

Product deduplication is a task where the solution depends entirely on the data: how many suppliers there are, how their exports differ, and which fields can even be matched at all. So the first step is a short review of your data, not a template proposal.

Send what you already have on hand: exports from two or three suppliers in their original form, a category and brand reference list if one is maintained, and a description of how a product currently gets into the catalog — manually, by script, or through an off-the-shelf module.

That is enough to see the field structure, estimate the share of duplicates, and understand which matching rules will work and which will produce false merges.

Next, we will review the data and come back with specifics:

• a field map for each supplier — where names, SKUs, barcodes, and units of measure match;
• an estimate of the volume of duplicates and typical discrepancies: typos, different word order, brand abbreviations;
• a proposal for matching rules — what we merge automatically, what goes to manual review;
• a queue scheme for moderating disputed pairs so a manager doesn't drown in manual decisions;
• an implementation plan with stages, timelines, and the points where you see results;
• rollback options in case a rule produces false positives on the live catalog.

The next step is a short conversation based on these materials. Describe the task in whatever form is convenient: how many suppliers, how often the exports are updated, who currently handles duplicates.

We will look at the data and come back with an estimate and a plan — what we do first, what delivers results fastest, and where your involvement will be needed.