AI contact deduplication: why duplicates distort CRM data
Duplicates in CRM do not appear because of a single mistake. The same organization is created from a website form, from a manager's email, from a call center export, and from integration with a billing system — and each card has its own set of fields.
The phone is recorded with and without the city code, the email has a typo in the domain, and the company name has three spelling variants.
Then the discrepancy in contact attributes multiplies across reports.
The funnel sees one deal on one card and loses the second as unfilled, the newsletter goes to the client twice and brings an unsubscribe, and the touch history is counted by a fragment rather than the client as a whole. Decisions are made based on numbers that do not exist in reality.
Manual merging of cards relies on a person: they open two records side by side, compare fields, and decide which value is correct.
On a base of tens of thousands of contacts, this means hundreds of hours of work and a constant risk of gluing together two different people with the same last names.
AI contact deduplication works differently. First, records are normalized — phones are brought to a unified format, company names and addresses are unified — then pairs are selected: not all records are compared with all, but only records with matching keys.
Next, the model evaluates candidates by a combination of fields, not by a single rule, and outputs a match probability: above the upper threshold, pairs are merged automatically; disputed ones go to a review queue with an explanation for each field.
The quality of the customer base becomes measurable: the share of duplicates falls to tenths of a percent, and each merge is logged together with the version of the master card — rollback remains possible. We explain how this is implemented and what will be required from your CRM.
What automatic duplicate merging in CRM gives
Automatic duplicate merging is work with the source of discrepancies, not with their consequences.
As long as one person has two or three cards in the base, analytics lies: deals are spread across managers, the history of requests is broken, the newsletter segment is doubled, and the client receives two emails with different offers.
The model compares records pairwise by normalized fields and finds matches where exact string comparison is powerless.
What the team records as a measurable result of implementation:
- Speed of base processing — instead of manually reviewing a sample, a stream works through the entire array, including archived cards and closed deals.
- Share of merged pairs — only disputed cases below the similarity threshold remain for the operator; they are a few percent of the total number.
- Data accuracy in CRM — a unified format for phone, address, and company name; during merging, the most complete field value is preserved, not the value from the old record.
- Reduction of manual operations — transfer of related objects (deals, tasks, correspondence) is performed according to rules, without manual re-attachment.
- A single master customer record — the card retains one identifier, duplicates go to the archive with a link to the main one, so rollback of the decision is possible.
Merge rules are configured for the schema of a specific base — from Bitrix24 to custom-built systems where some fields are maintained in free form. The output is a unified picture of the customer that can be relied upon in reports and segmentation.
Matching modes: rules, AI model, manual review
The choice of mode is a compromise between completeness and accuracy. Strict record matching rules give a predictable result on normalized data: phone, email, company domain.
But as soon as the name field contains “Vladimir” and “Vova”, and the city code is recorded differently, the rules stop seeing the pair.
The contact similarity model compares records by a combination of features and returns a match weight. The similarity threshold sets the boundary: above — we merge automatically, below — we leave as is.
Between these zones lies a gray area where confidence is insufficient and a human makes the decision.
| Mode | When it fits | Limitations |
|---|---|---|
| Strict rules | Directories with data-entry discipline, exchange with external systems | Does not catch typos, transliteration, different field formats |
| Similarity model | Bases after mass import, free-form input forms | Requires a configured threshold: at a low value, false pairs increase |
| Manual duplicate review | Expensive mistakes: large deals, legal contacts | Throughput is limited by people |
| Hybrid merge mode | Production bases where both coverage and accuracy are needed | Harder to implement: a cascade and a decision log are needed |
How this is assembled in practice
The hybrid merge mode builds a cascade: rules filter out exact matches, the model picks up fuzzy ones, manual review closes disputed pairs with a log of who confirmed what.
The threshold and feature weights are selected on your history — a labeled sample shows where the model makes mistakes, and the boundary is shifted to an acceptable balance.
Implementation stages: from base audit to handover
Duplicates in CRM are not cosmetic: the same organization comes from a website form, from an email, and from a call, and the manager sees three unrelated deal histories.
Therefore, we start not with configuration but with an audit — otherwise the merge rules will be either too soft or will glue together different clients.
- Customer base audit. We export contacts and deals, calculate the share of duplicates by segments, and see which channels they come from.
- Duplicate profile. We analyze exactly where records diverge: name spelling, phone format, Latin versus Cyrillic, empty fields. Here we also decide which features we consider reliable and which probabilistic.
- Matching scheme. An exact match of the key identifier leads to automatic merging, a fuzzy match on several fields goes to quarantine for manual review.
- Record merge configuration. We set the priority of sources for each field: take the phone from one system, the deal status from another, so that merging does not erase the interaction history.
- Testing on historical data. We run the rules on a copy of the base and calculate the share of false positives, not just the number of merges. We configure the confidence threshold before the automation touches the production base.
- Merge rollback. We log each merge with the record state before and after — an incorrect merge is rolled back, and the card returns to its original form.
- Handover. We provide the regulation, merge log, threshold values, and support instructions.
We separately walk through the worst-case scenario: a false merge brings two different clients into one card and confuses their deals, so the confidence threshold is fixed before enabling automation on the production base.
Matching rules, similarity thresholds, and merge regulations
AI-powered contact deduplication rules are not a setting inside a service but a working document that lives in the company.
After launch, the client retains a complete set: the described comparison logic, thresholds for each field, the conflict resolution procedure, and instructions for operators. Below is what exactly is recorded in this set.
- Matching rules: which fields participate in comparison (phone, email, name, company, address), in what order they are checked, and what is normalized before comparison — bringing the phone to a unified format, parsing abbreviations in organization names.
- Configured similarity thresholds separately for each field: strict matching for email and phone, fuzzy comparison for names and company names, indicating at which value a pair is merged automatically and at which it goes to manual review.
- Field priority policy: from which card the value is taken if records diverge — the rule “more complete record”, “record from a more reliable source”, “latest update”, or a combination of them by field types.
- Card merge regulations: what is transferred to the main record, what is archived, how the change history and related deals, tasks, and correspondence are preserved.
- Rules for handling disputed pairs: where ambiguous matches go, who reviews them and within what timeframe, how to roll back an erroneous merge.
- Integration documentation: CRM objects affected by the process, trigger events, rate limits on requests, and the procedure for re-running after a failure.
- A set of reference contact pairs used before any threshold change to verify that the logic still works on your data.
Case study: how automatic merging removed duplicates in a production base
Starting point
A base of 480 thousand contacts in amoCRM was filled from web forms, imports from exhibitions, and manual entry by managers, so the same person was present three times: “Ivan Petrov” (Cyrillic), “Ivan Petrov” (Latin), “Petrov I.” — with different phones and different deals.
A complete check showed a duplicate share of 9–11%: the manager saw only part of the correspondence for a client, calls went out repeatedly, and segment newsletters counted one person as three.
Manual review of such a base by operators would take months and still would not protect against new duplicates on the next import — we explain how this was closed.
How merge quality control was set up
First, a reference sample of pairs was assembled and labeled manually: identical records, different people with similar names, borderline cases with a shared work phone.
On it, comparison thresholds were calibrated — above the upper one, the pair goes to auto-merge; in the middle zone, it enters the operator queue with highlighting of differing fields; below, we do not touch it at all.
Each merge writes an entry to the log: which cards were merged, by what feature, with what confidence, and who confirmed. Deals, tasks, and correspondence are transferred, not deleted, so splitting by a log entry returns the cards to their original state.
Metrics before and after
The share of duplicates in the base fell from 9–11% to 0.3% and remains at this level: a nightly run picks up new pairs after each import.
The time for base processing decreased from months of manual reconciliation to a nightly run plus selective operator review of about 2% of matches. The number of erroneous merges is 0.1% of all merges, and each is rolled back in one operation using the merge history.
What to do if duplicates have already spread across deals and tasks?
Duplicates that have already accumulated deals, tasks, and correspondence cannot simply be deleted: the links would go away with the card, and a gap in revenue would appear in reports.
Merging with preservation of links means transferring dependencies to the main record, not deleting the extra card.
First, the system collects everything that references each of the cards: open and closed deals, tasks, calls, emails, invoices.
The main one is chosen where the history is more complete and the activity is more recent; the rest are marked as absorbed — this way, deals are not doubled in the funnel and the customer total is not distorted.
Correspondence history is not copied blindly during merging.
Messages are arranged into a single timeline, and emails that were in both cards are recognized by identifier and collapsed into one — otherwise the manager sees two identical client replies and does not understand which one is the latest.
A separate question is the person responsible for the client. If different employees are assigned to the duplicates, it is necessary to decide in advance who will manage the client further and whose tasks for them are closed.
Open tasks do not disappear with the card: they are re-attached to the main record so that work does not stall.
AI contact deduplication here works as untying a knot, not as garbage cleanup: the algorithm ranks candidates, builds a link map, and shows the merge plan before applying — with a list of deals, tasks, and emails that will move.
The original cards remain in the archive, so the result can be checked and, if necessary, disassembled manually.
Technical concerns: merging, rollback, history preservation
Merging cards is an operation that is scary to launch on a live base: one mistake glues two clients together, and the deal history goes to the wrong manager. Below are answers to the concerns heard before the start of almost every AI contact deduplication project.
What if the system makes a mistake and glues different people together?
The match threshold is set in advance and depends on the field: a complete match of email and phone means automatic merging; a partial one goes only through a confirmation queue by the base owner.
Disputed pairs are not changed until a human confirms, and the share of false positives is measured on a pilot sample before enabling automation.
Can a merge be rolled back?
Yes. Before merging, both original records are saved unchanged and marked with a common group identifier — the reverse operation returns the cards to their previous state along with their links. Rollback is available even a month later, not only at the moment of merging.
What happens to the contact history: calls, emails, tasks?
Related records are not overwritten but re-attached to the main card, so the chronology preserves order and authorship. A trace remains in the log: what was merged, by what feature, and when — a disputed case is resolved without restoring a backup.
Will the system go down, and who will get access to the data?
Processing runs in batches during low-activity hours, in portions with pauses — the load on the base remains within usual limits, and managers' work is not interrupted.
Rights are not expanded: merging is performed under a service account, and confirmation of disputed pairs is available only to the base owner.
Let's discuss the task: we will return with an estimate and work plan
To make the estimate realistic rather than “market average,” we need facts about your base.
Send a contact export in CSV — anonymized, without personal data: record identifier, source, creation and last modification dates, values of company, phone, email, and address fields as they are stored in the system.
This is enough to calculate the share of duplicates, analyze typical match scenarios, and estimate the volume of manual work.
It is useful to attach to the export:
- a link schema — deals, tasks, calls, emails, and attachments linked to contacts and companies;
- a list of record sources: forms, integrations, imports, mailboxes, telephony;
- the rules by which manual merging is currently done, and disputed cases;
- a list of required fields and requirements for change auditing;
- restrictions on access and data exchange mode;
- several examples of duplicates that you have already found.
On a call, we will clarify what is not visible from the export: which discrepancies to consider a duplicate, what to do with related records during merging, who confirms ambiguous pairs, and how to arrange rollback of an erroneous merge.
After that, we will return with an estimate and work plan: stages, project start order, composition of checks on a copy of the base, and the format of result handover — matching rules, run reports, and support instructions.
