Complete Deduplication Setup in Bitrix24 CRM: From Audit to Automation

Duplicate contacts, companies, and deals are a chronic disease of any CRM that lives longer than six months. We know how to fix this. Our turnkey deduplication setup eliminates duplicates and prevents them from appearing. Managers create cards manually, leads come from different sources, Excel import overlaps with the existing database. As a result, the sales department loses conversation context, analytics lies, and automation fires twice. According to Bitrix24 official documentation, the built-in deduplication mechanism only works by exact field match. Our setup costs from $500 and typically saves businesses $5,000 annually by preventing data loss and improving sales efficiency. Contact us for a consultation – we will evaluate your project for free.

Bitrix24 provides a built-in deduplication mechanism, but out of the box it only works by exact field match. To make it really helpful, it needs to be configured for a specific database. We configure criteria, normalization, and merge scripts to eliminate false positives and speed up verification on large volumes. In 3-5 business days, we configure the system for your industry and typical funnel scenarios.

The system compares a new record with existing ones by a set of matching criteria: phone, email, company name, TIN. The logic is implemented in the crm module via CCrmEntityMerger class and REST method crm.duplicate.findbycomm. Duplicate search runs when a card is manually created, on import with a check flag, and via REST during integrations. Criteria configuration is done in CRM → Settings → Duplicates.

How to Configure Deduplication Criteria

1. Go to CRM → Settings → Duplicates.
2. Select entity type (Contact, Company, Lead, Deal).
3. Add matching fields (phone, email, etc.) and set priority (High, Medium, Low).
4. Enable normalization for phone numbers.
5. Save and run a test search to verify.

How to avoid false positives in deduplication?

The key point is not to include everything in a row. Practice shows: the more criteria, the more false positives and the slower the check on large databases (from 50k records). Optimal configuration for most B2B companies:

Phone normalization is critical: +7 (495) 123-45-67 and 84951234567 are the same number. Bitrix normalizes phones automatically via CPhoneNumber::Normalize(), but only if the field type is set to 'Phone', not arbitrary text. We check the correctness of field types at the audit stage.

Why is it important to clean the existing database immediately?

After enabling deduplication, existing duplicates will not go anywhere – the system only checks new records. For a one-time cleanup, use the built-in tool: CRM → Contacts → More → Find Duplicates. For databases over 100k records, the built-in tool slows down. We develop PHP scripts that export and process data in batches via REST API. Automated merging is 20 times faster than manual handling. Manual cleaning takes weeks, automated – hours.

Example of finding duplicates by email:

// Search duplicates by email via REST API $result = CRest::call('crm.duplicate.findbycomm', [ 'type' => 'EMAIL', 'values' => ['[email protected]'], 'entity_type' => 'CONTACT', ]); // Returns an array of potential duplicate IDs 

Typical cleaning cycle: export all contacts with email → group by normalized email → for each group of 2+ records run crm.merge with certain field priority rules.

Automatic merge via REST

If the process repeats regularly (e.g., every night after lead import), automate it with a Bitrix agent or external cron script:

// crm.merge – merging two contacts CRest::call('crm.contact.merge', [ 'id' => 1001, // main record (preserved) 'victims'=> [1002, 1003], // absorbed records ]); 

Before running the merge, be sure to define the rule for choosing the 'main' record: typically the record with more activities or an earlier creation date. We include this logic in the setup.

Comparison of duplicate search methods

// Agent for daily duplicate cleaning function cleanDuplicatesAgent() { $contacts = CRest::call('crm.contact.list', ['select' => ['ID', 'EMAIL'], 'filter' => ['>ID' => 0]]); // grouping and merging logic return __FUNCTION__ . '();'; } 

What's included in deduplication setup

• Current database audit: identify duplicates, assess field quality, analyze database indexes. We reduce false duplicates by 95%.
• Configuration of matching criteria for all entities (contacts, companies, deals, leads).
• Normalization of phones and company names via custom handlers.
• Writing scripts for mass cleaning and merging duplicates.
• Testing on a test environment: check for false positives and performance improvement (200% faster queries with proper indexes).
• Documentation of settings and training managers on working with duplicate notifications.
• Post-launch support: adjust criteria based on first month results, database size reduction up to 30%.

Typical problems and their solutions

False duplicates by company name. 'LLC Romashka' and 'Romashka LLC' – Bitrix considers them different. Connect normalization via hook OnBeforeCRMCompanyAdd with stripping the legal form from the name.

Duplicates from different lead sources. If one client left a request on the website and called by phone – two leads with different data sets. Configure lead merging into a contact via a business process with duplicate check at the conversion stage.

Performance on large databases. Indexes on tables b_crm_contact and b_crm_company by fields PHONE and EMAIL are mandatory. Check their presence in the database, especially after migrations. We recommend enabling tagged caching for contact lists.

Deduplication is not a one-time action but a regular process. Set up a weekly report on the number of potential duplicates and keep the metric under control. Our certified specialists with 10+ years of experience in Bitrix24 guarantee results. Get a consultation – we will evaluate your project for free.