Automated Deduplication of CRM Contacts Using Machine Learning

Automated Deduplication of CRM Contacts Using Machine Learning

In a CRM with 100,000 contacts, every fifth one is a duplicate. The same customer is entered three times: as "John Smith", "Smith John", and "[email protected]". The result: analytics lie, emails go to spam, managers spend hours on "new" leads that are already customers. We solve this problem with a combination of three methods: rule-based, ML, and embedding. After implementation, the database becomes clean—duplicate percentage drops from 20% to 2-5%, saving your company $30,000 annually. We have completed deduplication on 50+ projects with over 50 million records.

Entity Resolution is a classic task, but in CRM there is specific complexity: fields are filled irregularly, with transliteration and typos. Simple exact matches cover only 40% of duplicates. So we build a multi-layered system using probabilistic matching and cosine similarity for embeddings.

What Problems We Solve

• Bloated database: duplicates take up space and distort analytics. A typical CRM with 100,000 contacts contains 10-25% duplicates. Reducing the database by 8-15% saves on storage and campaigns – for a mid-sized company, this can exceed $50,000 annually.
• Communication failures: a customer receives three identical emails—marks as spam and unsubscribes. After deduplication, the unsubscribe rate drops by 30%, retaining up to 15% of the marketing budget.
• Sales mistakes: a manager spends time on a "new" lead that is already an existing customer. Time loss—up to 20 hours per month for a team of 10.

How AI Finds Duplicate Contacts

We use three detection layers:

1. Rule-based (fast filtering): exact email or phone match—a confident duplicate. Accuracy 99%, but low recall (about 40%).
2. ML model (entity resolution): the dedupe library—trained on labeled pairs via active learning. Handles typos, transliteration, and missing fields. Accuracy 92-95%, recall 80-85%.
3. Embedding-based (scaling): we convert each contact into a vector (all-MiniLM-L6-v2, 384-dimensional embeddings) and find nearest neighbors via faiss. Processes millions of records in seconds, accuracy 88%, recall 85%.

Comparison of methods:

The ML model is 1.5 times more accurate than the rule-based approach and 2 times faster than an embedding-only approach for databases up to 500k records.

ML Deduplication Model (Code)

import pandas as pd import dedupe from dedupe import Dedupe class ContactDeduplicator: def __init__(self): self.deduper = None def setup_fields(self): """Describe fields for dedupe""" fields = [ dedupe.variables.String('first_name'), dedupe.variables.String('last_name'), dedupe.variables.String('email', has_missing=True), dedupe.variables.String('phone', has_missing=True), dedupe.variables.String('company'), dedupe.variables.String('job_title', has_missing=True), ] return dedupe.Dedupe(fields) def train(self, records: dict, training_file: str = None): """Train on labeled pairs (match/not-match)""" self.deduper = self.setup_fields() if training_file and os.path.exists(training_file): with open(training_file) as f: self.deduper.prepare_training(records, f) else: self.deduper.prepare_training(records) # Active learning: label example pairs dedupe.console_label(self.deduper) with open(training_file, 'w') as f: self.deduper.write_training(f) self.deduper.train() def find_duplicates(self, records: dict, threshold: float = 0.5) -> list[tuple]: """Find duplicates with probabilities""" clustered_dupes = self.deduper.partition(records, threshold) duplicate_groups = [] for (cluster_id, record_ids, scores) in clustered_dupes: if len(record_ids) > 1: duplicate_groups.append({ 'records': list(record_ids), 'scores': list(scores), 'max_score': max(scores) }) return sorted(duplicate_groups, key=lambda x: x['max_score'], reverse=True) 

Fuzzy String Comparison

from rapidfuzz import fuzz, process def compute_similarity(record1: dict, record2: dict) -> float: scores = [] # Email: exact or domain match if record1.get('email') and record2.get('email'): if record1['email'].lower() == record2['email'].lower(): return 1.0 # Exact email match — definitely a duplicate email1_domain = record1['email'].split('@')[1] email2_domain = record2['email'].split('@')[1] if email1_domain == email2_domain: scores.append(0.5) # Same domain — similar # Name: fuzzy match name1 = f"{record1.get('first_name', '')} {record1.get('last_name', '')}" name2 = f"{record2.get('first_name', '')} {record2.get('last_name', '')}" name_score = fuzz.token_sort_ratio(name1, name2) / 100 scores.append(name_score * 0.4) # Phone: normalize and compare phone1 = re.sub(r'\D', '', record1.get('phone', '')) phone2 = re.sub(r'\D', '', record2.get('phone', '')) if phone1 and phone2: if phone1[-10:] == phone2[-10:]: # Last 10 digits scores.append(0.9) # Company if record1.get('company') and record2.get('company'): company_score = fuzz.token_set_ratio( record1['company'], record2['company'] ) / 100 scores.append(company_score * 0.2) return sum(scores) / len(scores) if scores else 0.0 

Record Merge Strategy

def merge_duplicates(records: list[dict]) -> dict: """Merge a group of duplicates into one record""" merged = {} field_priority = ['email', 'phone', 'first_name', 'last_name', 'company'] for field in field_priority: values = [r.get(field) for r in records if r.get(field)] if not values: continue # Take the most frequent value merged[field] = max(set(values), key=values.count) # For created_at, take the earliest date dates = [r.get('created_at') for r in records if r.get('created_at')] if dates: merged['created_at'] = min(dates) # Merge tags and labels all_tags = [] for r in records: all_tags.extend(r.get('tags', [])) merged['tags'] = list(set(all_tags)) merged['merged_from'] = [r['id'] for r in records] return merged 

Why Implement ML-Based Deduplication?

Rule-based misses typos and transliteration. Embedding-based without fine-tuning gives false positives. The ML model on dedupe is the sweet spot: it trains on your data in a few hours of active labeling, with 92-95% accuracy and 80-85% recall. We guarantee at least a 10% reduction in duplicate percentage—proven on 50+ projects. The project cost is calculated individually based on data volume and integration complexity. A typical project for 50,000 records costs $4,000 and yields $20,000 annual savings. A recent project for a retail client with 200k records cost $10,000 and resulted in $40,000 annual savings from reduced marketing waste and improved sales efficiency.

Work Process

1. Database audit—we export contacts, assess current duplicate percentage.
2. Strategy selection—rule-based + ML or embedding for large volumes.
3. Labeling and training—prepare training set, train model via active learning with human-in-the-loop.
4. Integration—API or direct access to CRM (Bitrix24, AmoCRM, Salesforce).
5. Testing—A/B comparison: automatic merge vs manual audit. We compute precision, recall, and F1 score.
6. Deployment and monitoring—periodic deduplication pipeline with anomaly alerts. We use confusion matrices to track performance.

Timeline for different data volumes:

What's Included

• Documentation: model description, threshold settings, instructions for retraining.
• Access: to source code (GitLab), trained model (MLflow), metrics dashboard.
• Training: a session for analysts (how to label new data).
• Support: 1 month—bug fixes, threshold tuning.

Contact us for a free audit of your CRM—we'll assess the duplicate percentage and economic impact. Reach out through the form on the website or via phone.