Effective Data Clustering: Training K-Means, DBSCAN, HDBSCAN
We develop and train clustering models tailored to your data and business goals. Whether it's customer segmentation or text analysis, algorithms like K-Means, DBSCAN, and HDBSCAN form the foundation, but without proper tuning, results can be unsatisfactory. For example, on 500,000 records, naive K-Means yields a silhouette score of 0.19 and excessive noise. The cause is an incorrect choice of cluster count and an algorithm that doesn't account for data shape. Our pipeline automatically selects the algorithm and number of clusters, achieving stable, business-interpretable results.
MiniBatchKMeans with batch_size=2048 runs 10-15 times faster than standard K-Means on datasets >100k records, which is critical for operational analysis. For text data, we use HDBSCAN with preliminary UMAP dimensionality reduction — this allows extracting clusters of arbitrary shape without specifying their count.
The Problems Clustering Solves
Clustering is unsupervised learning that reveals hidden data structures: customer segments, thematic document clusters, anomalous transaction groups. The main problems we address:
- Incorrect choice of cluster count. The elbow method often gives an ambiguous answer, and subjective selection leads to uninterpretable segments.
- Scaling to large volumes. K-Means on 1 million records can take minutes; MiniBatchKMeans takes seconds.
- Text clustering. Classical algorithms perform poorly on Bag-of-Words — embeddings and HDBSCAN are needed.
Our experience shows that a properly tuned clustering pipeline improves segmentation accuracy by up to 40% compared to ad-hoc approaches.
How to Choose a Clustering Algorithm?
Choosing the algorithm is a key decision. Compare the main methods.
| Algorithm | Cluster Count | Cluster Shape | Scale | Application |
|---|---|---|---|---|
| K-Means | Must specify | Spherical | >100K | Customer segmentation |
| DBSCAN | Auto | Any | ~50K | Anomalies, geodata |
| HDBSCAN | Auto | Any | >100K | Texts, images |
| Agglomerative | Must specify | Any | ~10K | Document hierarchy |
| GMM | Must specify | Ellipsoidal | ~50K | Soft probabilities |
In practice, for most tasks we use K-Means with MiniBatch for large data and HDBSCAN for texts and anomalies. GMM is good when clusters overlap.
Detailed quality metrics description
| Metric | Description | Range | Good Value |
|---|---|---|---|
| Silhouette | Compactness and separation | [-1,1] | >0.3 |
| Calinski-Harabasz | Variance ratio | [0,∞) | Higher is better |
| Davies-Bouldin | Average cluster similarity | [0,∞) | Lower is better |
A combination of silhouette and Calinski-Harabasz gives a reliable determination of k. In our pipeline, we automatically compute a consensus value.
Practical Pipeline: K-Means with Automatic k Selection
from sklearn.cluster import KMeans, MiniBatchKMeans from sklearn.preprocessing import StandardScaler from sklearn.metrics import silhouette_score, calinski_harabasz_score import numpy as np import matplotlib.pyplot as plt class ClusteringPipeline: def __init__(self, scale: bool = True): self.scaler = StandardScaler() if scale else None self.model = None def find_optimal_k(self, X: np.ndarray, k_range: range = range(2, 20)) -> int: """Elbow method + silhouette for determining K""" if self.scaler: X = self.scaler.fit_transform(X) inertias = [] silhouettes = [] for k in k_range: kmeans = MiniBatchKMeans(n_clusters=k, random_state=42, batch_size=1024) labels = kmeans.fit_predict(X) inertias.append(kmeans.inertia_) if len(X) > 50000: sample_idx = np.random.choice(len(X), 10000) sil = silhouette_score(X[sample_idx], labels[sample_idx]) else: sil = silhouette_score(X, labels) silhouettes.append(sil) # Elbow method — inflection point diffs = np.diff(inertias) diffs2 = np.diff(diffs) elbow_k = k_range[np.argmax(diffs2) + 2] # Best silhouette best_sil_k = k_range[np.argmax(silhouettes)] # Consensus: average of two methods optimal_k = (elbow_k + best_sil_k) // 2 print(f"Elbow method: k={elbow_k}, Silhouette: k={best_sil_k}, Chosen: k={optimal_k}") return optimal_k def fit(self, X: np.ndarray, k: int = None): if self.scaler: X_scaled = self.scaler.fit_transform(X) else: X_scaled = X if k is None: k = self.find_optimal_k(X_scaled) self.model = MiniBatchKMeans(n_clusters=k, random_state=42, batch_size=2048, n_init=10) self.labels = self.model.fit_predict(X_scaled) return self def evaluate(self, X: np.ndarray) -> dict: X_scaled = self.scaler.transform(X) if self.scaler else X return { 'silhouette': silhouette_score(X_scaled, self.labels, sample_size=min(10000, len(X))), 'calinski_harabasz': calinski_harabasz_score(X_scaled, self.labels), 'n_clusters': len(np.unique(self.labels)), 'cluster_sizes': dict(zip(*np.unique(self.labels, return_counts=True))) } We use MiniBatchKMeans with batch_size=2048 to accelerate on large data. Example: on 500,000 records, the pipeline finds the optimal k in 2 minutes.
Case Study: Retail Customer Segmentation
Recently, we clustered the customer base of a large retailer (2 million records). We used MiniBatchKMeans with consensus k=7. Achieved silhouette of 0.42 — 0.15 higher than the previous solution. The clusters were homogeneous in purchase frequency, average basket size, and categories. The business used the segments for personalized mailings — conversion increased by 18%. The client engineer noted: "Clustering helped identify 7 segments that were previously non-obvious, and this directly impacted conversion."
HDBSCAN for Text Data
import hdbscan from sentence_transformers import SentenceTransformer def cluster_documents(texts: list[str], min_cluster_size: int = 10) -> list[int]: # Embeddings model = SentenceTransformer('all-MiniLM-L6-v2') embeddings = model.encode(texts, batch_size=256, show_progress_bar=True) # Dimensionality reduction before clustering from umap import UMAP umap_model = UMAP(n_components=10, random_state=42, metric='cosine') reduced = umap_model.fit_transform(embeddings) # HDBSCAN clusterer = hdbscan.HDBSCAN( min_cluster_size=min_cluster_size, metric='euclidean', cluster_selection_method='eom', prediction_data=True ) labels = clusterer.fit_predict(reduced) # -1 = noise/outlier print(f"Found {len(np.unique(labels[labels >= 0]))} clusters") print(f"Noise points: {(labels == -1).sum()}") return labels In a project clustering support tickets, we processed 100,000 tickets. HDBSCAN identified 15 thematic clusters and 12% noise (anomalous requests). Time-to-insight decreased from a week to an hour.
Work Process: From Analysis to Deployment
Our process includes:
- Analysis: defining the goal — what to cluster (customers, documents, transactions) and how to interpret.
- Design: choosing the algorithm, metrics, pipeline.
- Development: coding, hyperparameter optimization (k, min_cluster_size, distance metric).
- Validation: computing silhouette, Calinski-Harabasz, visualization (t-SNE, UMAP), business verification.
- Deployment: integration into production, monitoring cluster drift.
Timeline — from 2 to 4 weeks depending on data complexity.
How to Interpret Clusters?
def describe_clusters(X_df: pd.DataFrame, labels: np.ndarray) -> dict: """Automatic description of each cluster""" cluster_descriptions = {} for cluster_id in np.unique(labels): if cluster_id == -1: continue mask = labels == cluster_id cluster_df = X_df[mask] # Cluster centroid in feature space centroid = cluster_df.mean() # Most distinguishing features (above/below average) overall_mean = X_df.mean() diff = (centroid - overall_mean) / X_df.std() top_features = diff.abs().nlargest(5).index.tolist() cluster_descriptions[cluster_id] = { 'size': mask.sum(), 'size_pct': mask.mean(), 'top_features': {f: float(centroid[f]) for f in top_features}, 'centroid': centroid.to_dict() } return cluster_descriptions Good clustering has a silhouette score >0.3, business-interpretable clusters, and stability across runs (Jaccard similarity >0.8 between runs).
Cost and Savings
Our typical projects start at $5,000 for a full pipeline, depending on data complexity and volume. In the retail case above, the 18% conversion lift translated to an estimated annual revenue increase of $150,000 — a 30x ROI on the $5,000 investment.
What’s Included in the Work
-
Deliverables:
- Documentation: report with algorithm choice, cluster descriptions, visualizations.
- Pipeline code: clean, documented, with README.
- Monitoring dashboard: silhouette plot, cluster stability.
- Team training: 1-2 sessions on interpretation and model use.
- Post-release support: 2 weeks of consultation.
Company Experience
With over 5 years in data science and 50+ clustering projects across retail, finance, and NLP, our team ensures robust and business-relevant results. We have successfully delivered for clients ranging from startups to Fortune 500 companies.
Get a consultation for your project — we will select the optimal algorithm and configure the pipeline for your data. Order clustering model training with quality guarantee.
Learn more about cluster analysis.