Why duplicates accumulate in CRM and how it gets in the way
Duplicates appear not because managers are careless, but because of the architecture of the inbound flow.
The same client comes in through a website form, an email to a shared inbox, a phone call, and a request in a messenger — and each channel creates a new card because it does not check against the existing database.
On top of that come imports from exports, manual entry during parallel work of several departments, and old records left over after a change of telephony or email provider.
The cost of such a "dirty" database is not immediately visible, but it shows up in three places. A manager calls a client who has already bought from a colleague and spoils the impression.
Funnel analytics counts one deal twice, and conversion by channel stops reflecting reality. Segmentation for mailings gets inflated: instead of one clean record, the selection includes three or four fragments of history, and the client receives an email they have already seen.
Merging duplicates in Bitrix24 solves this at the data level, not at the level of persuading people to "fill in the fields more carefully." The system looks for matches by normalized keys — phone, email, company name taking the legal form into account — and then the cards are merged into one while preserving the entire history: calls, emails, deals, tasks, and comments.
Nothing is lost, duplicate contacts and duplicate companies disappear from the database, and the search rules are configured for the real scenarios of a specific sales department.
A separate task is to prevent re-accumulation. Automatic checking when a lead is created, normalization of incoming data, and a policy for imports keep the client database clean on an ongoing basis, not just once. CRM data quality stops depending on the human factor — let's break down how to build this.
What a clean customer base delivers
Merging duplicates is not a one-time cleanup, but a change in how data behaves going forward. When a customer has a single record, the entire history — calls, emails, invoices, deals — comes together in one place.
The manager stops switching between records and spends time selling rather than searching for which record holds the needed contract.
The speed of work grows predictably: instead of several records with fragments of history, there is a single customer record showing the entire deal journey.
This eliminates manual reconciliations and repeat calls — the manager immediately sees what stage the customer is at and what has already been promised.
The accuracy of CRM reports directly depends on how many records fall to a single customer. While duplicates exist, revenue is double-counted, the funnel shows non-existent stages, and conversion is calculated on distorted data.
After merging duplicates, reports begin to reflect the real picture: one deal — one row, one customer — one history.
A separate effect is the reduction in the number of lost deals. Duplicates hide active deals in records that no one is working: the customer is ready to buy, but their request has settled in a forgotten record. Merging surfaces such deals and brings them back into play.
Finally, order in the database reduces the load on the system itself: fewer records — faster search, more stable exports, fewer conflicts when syncing with 1С. The database stops growing chaotically, and it is easier to keep it in working order.
Methods for merging duplicates: manual, batch, automatic
The choice of mode is determined not by interface convenience, but by the size of the database and the quality of the data. Manual duplicate merging is opened by an operator: they see the cards side by side, decide for themselves which fields to keep, and confirm the merge.
This is the only mode where disputed cases — different people with similar names, different companies within the same holding — are separated deliberately, without false merges.
Batch merging works on a selection: we filter the list by a matching rule, mark the found pairs, and launch mass processing of the database.
Automatic merging is tied to events — lead creation, import, web form — and merges matches on input, without waiting for the database to grow. Below is how the modes differ in terms of application conditions and where each one has its limit.
| Mode | When it fits | Limitations |
|---|---|---|
| Manual | Small database, heterogeneous data, disputed matches | Operator speed, does not scale to thousands of records |
| Batch | Accumulated duplicates, one-time cleanup before migration or reporting | Requires verified rules, otherwise it will merge extra records; result control is needed |
| Automatic | Constant flow of requests from forms, email, telephony | False triggers with weak rules; a merge log and rollback are mandatory |
Matching rules are the common denominator for all three modes: a match by phone, email, company details, or a "name + company" combination. The stricter the rule, the fewer false merges, but the higher the share of missed duplicates.
In practice, the modes are combined: automation handles the flow, a batch run cleans up what has accumulated, and manual mode closes out disputed pairs.
How duplicate merging is implemented
Implementation starts not with configuration, but with a database audit: we export all CRM entities — leads, contacts, companies, deals — and count how many records actually fall under Bitrix24 duplicate merging.
At this stage, it becomes clear which fields are matched (phone, email, company name, details), where normalization failed, and why duplicates appeared in the first place: manual entry, import from 1C, web forms, email integrations.
Next, we design the merge rule scheme — which fields are considered key, which value wins in a conflict, and what to do with related deals, tasks, and call history.
We document the rules in writing: without this, in a month no one will remember why a particular phone number was chosen for two cards.
- CRM database audit. Exporting entities, assessing the volume of duplicates, analyzing the sources of their appearance.
- Merge rule scheme. Field priority, conflict logic, the fate of related records and history.
- Merge configuration. Configuring the rules in Bitrix24, and if necessary, customizing the logic for non-standard scenarios.
- Test run. Merging on a database copy or a limited sample, comparing the result with expectations.
- Transfer to the production database. Phased merging with control at each step and the ability to roll back.
- Handover of regulations. Instructions for managers: how to check a card before creating it, how to respond to duplicates.
- Support. Monitoring the database after implementation, adjusting the rules as new sources appear.
The test run is a mandatory stage, not a formality: it shows how many records will merge correctly and where the rules produce false positives.
After the transfer, we hand over the regulations to the team and stay in touch — merge rules have to be fine-tuned when new channels for incoming requests appear in the CRM.
Matching rules and data protection policies
Matching rules are configured in the module settings and determine which fields the system uses to consider records as a single client. Usually these are a normalized phone number, email, and company details — a tax ID or name, taking into account the legal form.
Normalization is mandatory: the phone is brought to a unified format with a country code, email is lowercased, and quotes, extra spaces, and abbreviations such as "LLC" and "LLC "" in various spellings are removed from names.
Without this, "+7 (999) 123-45-67" and "89991234567" will remain different contacts.
Field priority determines whose data ends up in the final card. The logic is this: company details and phone are considered more reliable than an email or a manually entered name; a value from a field with higher priority overwrites a less reliable one.
For each attribute, the behavior in case of a conflict is configured — keep the value from the main card, take it from the duplicate, or save both as alternatives. This protects against a situation where, after merging, a work number is lost and an outdated one remains.
Protection against erroneous merging is built on a match threshold and manual confirmation. The system does not merge records automatically if only one attribute matches: two different people with the same name are not a duplicate.
Bulk merging is launched on a schedule with a limit on the number of operations and a mandatory report, while disputed pairs go into a separate queue for review by the responsible person.
Access policies delineate who sees duplicates and who has the right to merge them. The right to merge is usually assigned to an administrator or department head, while regular employees work only with their own contacts.
All operations are written to a log: who, when, and which cards were merged — this makes it possible to roll back an erroneous merge and investigate the incident.
Bitrix24 stores the change history for each card, so recovery after an incorrect merge takes minutes rather than hours of manual entry.
What the client receives after the work
After setting up duplicate merging in Bitrix24, the client receives not a one-time database cleanup, but a working mechanism that continues to cut off duplicates without manual intervention. Below is what exactly remains in the project after handover.
- Merge rules scheme — which fields are compared (phone, email, company details, company name), which are considered priority in a conflict, which exceptions are set for branches and partner contacts.
- Configured duplicate merging rules — automatic scenarios for leads, contacts, and companies, including trigger conditions and the match threshold at which a record goes into merging rather than manual review.
- Merging regulations — a document describing the procedure: who initiates the merge, which records are considered master records, what to do with linked deals, tasks, and call history.
- Merge log — a report showing which records were merged, when, and by which rule. It allows you to investigate a disputed situation and confirm that history has not been lost.
- Instructions for managers — a short practical document: how to recognize a duplicate, how to start a merge manually, where to look in case of a conflict, and which fields not to touch.
- Database documentation — a description of the field structure, relationships between entities, and storage logic, so that new employees and contractors understand how the CRM is organized.
- Set of checks and reporting — regular monitoring of duplicate growth, so that the rules do not become outdated when request sources change.
Each artifact is handed over in working form: the rules are already in effect, the log is being populated, and the instructions are stored in the team's knowledge base. Describe your task — we will get back to you with an estimate and a work plan.
How automatic merging differs from manual merging
Automatic merging works according to rules you set in advance: a match by phone, email, company name, or a combination of fields.
When a record enters the system, the rule compares it against the database and either merges the entries or flags it as a potential duplicate for review. Manual merging is when a manager opens two records themselves, selects the primary one, and transfers the data. The difference is not in speed, but in who makes the decision in a disputed situation.
Automation wins where the match is unambiguous: the same phone number, a normalized email, an exact company name. Here manual review only slows things down — the manager spends time on obvious pairs while the lead waits for a response.
The rule triggers in seconds, and the pipeline doesn't get overgrown with copies. But as soon as fuzzy matches appear — similar names, a shared corporate phone number, different spellings of the same client — automation starts making mistakes.
The main risk of automation is merging two different deals or losing the correspondence history when the wrong primary record is selected. That is why there is always a threshold in the setup: strict rules merge immediately, soft ones only suggest candidates.
Quality control is built on a merge log: you can see what was merged and by which rule, and the entry can be split back apart if the rule triggered falsely.
Choosing a mode is not either-or, but a combination. We hand strict matches to automation, and disputed pairs go to a responsible person for manual review.
This hybrid covers both speed and the risk of error: routine duplicates resolve themselves, while decisions on ambiguous clients remain with a human.
Case study: how duplicates in the customer database were reduced
The client is a wholesale company with a database of about 180,000 contacts accumulated over several years from website forms, calls, email, and uploads from 1C.
Managers complained that for a single customer, three or four deals were visible in the cards, the correspondence history was broken, and repeat inquiries were getting lost.
Our task was not just to "clean up the database" but to establish a rule under which duplicates stop appearing again.
What it was like before implementation
From the export, we counted 41% of cards with overlaps by phone, email, or company name. Some duplicates arose at the entry point: the website form created a lead even if the contact was already in the database, and the upload from 1C added yet another record with the same INN.
As a result, funnel analytics diverged from reality, and managers spent time on manual reconciliation instead of working with the customer.
What was changed
We set up merge rules in the CRM Bitrix24: priority by INN and phone, with an additional check by normalized company name and email domain.
For bulk processing, we ran the database through batch merging with an operations log — each pair of cards indicating which one remained the primary one and which fields were pulled in.
Separately, we closed off the source: the form and the upload from 1C now first look for a match and update the existing card rather than creating a new one.
Result for the business
The share of duplicates in the database fell from 41% to 6% — the remainder are deliberate exceptions where the companies really are different. The number of active deals in reports decreased by a third, but the figures now matched actual sales.
Managers stopped spending 15-20 minutes a day on manual reconciliation, and repeat inquiries are now visible in a single card with the full history.
The rule works at the entry point, so the database no longer becomes overgrown with duplicates — this is the main result of the implementation practice.
Common technical concerns
Before enabling merging on a production database, the technical stakeholder usually checks three things: what will happen to the deal history, how field conflicts will be resolved, and whether the operation can be rolled back.
Below is an analysis of these concerns at the level of Bitrix24 mechanics, rather than general reassurances.
Are deal history and activity lost during merging?
No, if the merge is configured correctly. The duplicate merging mechanism transfers related entities — deals, contacts, calls, emails, tasks, and timeline records — to the primary card rather than deleting them.
The key point: before the transfer, it is recorded which card is designated as the primary one. Everything linked to the absorbed record moves over to it while preserving dates and authors.
Loss is only possible if a duplicate is manually deleted instead of merged — these are different operations, and we always keep them separate in the procedure.
What should be done if fields in duplicates conflict?
Field conflicts are resolved by source priority. For each field, a rule is defined: the value from the primary card, the value from the absorbed one, or a merge (for example, combining phone numbers and emails into multi-fields).
Automatic merging without rules can indeed overwrite a more complete card with a less complete one — that is why we first export a slice of duplicates, run it on a copy of the portal, and only then configure priorities. This way, field conflicts stop being a lottery.
Can a merge be rolled back if something goes wrong?
There is no standard rollback of a merge in the interface — this is a deliberate limitation, not an oversight. That is why merge safety is built on preparation: before a mass operation, a backup of the portal is made and the cards subject to merging are exported.
Data recovery is then performed from the backup in a targeted manner, without rolling back the entire portal. For regular cleanup, we set up an operations log — it shows which cards were merged, when, and by which rule.
If any doubts remain regarding a specific portal — describe the current field structure and the volume of duplicates, and we will come back with an estimate and a safe merge plan.
Let's discuss your task of merging duplicates
To come back with an estimate and a plan for merging duplicates in Bitrix24, we need not a general request but a snapshot of your database: how many contacts and companies, which fields most often have divergent records, and whether there are integrations that create contacts automatically.
Usually an export or access to a test environment is enough — after that we calculate the volumes and risks ourselves.
What we clarify at the start:
- database size: the number of contacts, companies, deals, and leads, the share of records without a phone and email;
- sources of duplicates: website forms, mailboxes, telephony, imports from 1C or external services;
- merge rules: which fields we consider key, what to do when values conflict, which records must not be touched;
- related entities: deals, tasks, emails, and calls linked to duplicate cards;
- access: read and write permissions, a test environment for running the merge before the production database;
- constraints: required fields, business processes, and robots that trigger when a card is changed.
Based on this data, we audit the database, show how many duplicates are actually found and by which criteria, and propose a merge order — from safe groups to disputed ones.
You receive an estimate and a plan: the stages of work, what we do in the test environment, how we verify the result, and at what point we move the changes to the production database.
Describe the task — which fields you consider key and what you have already tried — and we will come back with an estimate, a plan, and a list of next steps.