Note: when tens of thousands of unstructured documents accumulate in a corpus – customer requests, scientific articles, legal contracts – manual categorization becomes impossible.
Clustering groups documents by semantic similarity without a single label, using embeddings and machine learning algorithms.
The main difficulty is the curse of dimensionality: direct 768-dimensional vectors from Sentence-BERT produce poor clusters without dimensionality reduction.
We implemented a pipeline for three large projects: from technical support segmentation (50 thousand dialogues) to organizing a corporate archive (200 thousand contracts). Stack: cointegrated/rubert-tiny2, UMAP, HDBSCAN.
In this article, we will break down each stage, compare algorithms, and show how to achieve Silhouette Score >0.4. We will also cover typical mistakes: ignoring stop words, incorrect min_cluster_size, and using only one metric.
This allows saving up to 40% of time on manual labeling. Get a consultation for your corpus – we will assess the applicability of clustering.
Clustering Pipeline
The canonical pipeline we use looks like this:
- Cleaning and normalization – removal of garbage characters, lemmatization (pymorphy2 or Mystem), stop-word filtering.
- Embeddings – converting texts into fixed-dimensional vectors (Sentence-BERT, ruBERT).
- Dimensionality reduction – UMAP compresses 768-dimensional vectors to 10–50 dimensions while preserving topology.
- Clustering – HDBSCAN or K-Means.
- Interpretation – TF-IDF top words or LLM summary.
- Visualization – 2D map via t-SNE/UMAP, color labeling of clusters.
Example configuration for HDBSCAN
import hdbscan from umap import UMAP reducer = UMAP(n_components=15, metric='cosine') clusterer = hdbscan.HDBSCAN(min_cluster_size=10, min_samples=1) How to Choose a Clustering Algorithm?
Algorithm choice is a key trade-off between speed and flexibility.
| Algorithm | Strengths | Weaknesses | When to use |
|---|---|---|---|
| K-Means | Millions of points, linear complexity | Requires K, sensitive to outliers | Categories known in advance (e.g., 10 types of requests) |
| HDBSCAN | Does not require K, automatically detects noise | Slower on large data without optimization | Exploratory analysis, unknown number of topics |
| BERTopic | End-to-end from text to topics, LLM interpretation | Depends on embeddings, resource-intensive | Rapid prototyping with visualization and doc templates |
For a typical project with 10 thousand documents, we choose HDBSCAN – it provides robust clusters and automatically filters out noise. If instant interpretation is needed, we add BERTopic.
Comparison of Embedding Models
| Model | Dimensionality | Language | Speed | Quality |
|---|---|---|---|---|
cointegrated/rubert-tiny2 |
312 | RU | high | medium |
sbert-base-ru-mean-tokens |
768 | RU | medium | good |
text-embedding-3-small (OpenAI) |
1536 | Multi | high | excellent |
How We Do It: Technical Support Request Segmentation Case
Task: divide 50,000 customer dialogues from the last 3 years into thematic clusters. There is no labeling.
Stack: cointegrated/rubert-tiny2 (quality/speed trade-off), UMAP -> HDBSCAN. After clustering, each cluster was interpreted via TF-IDF top-10 words and additionally via GPT-4o, passing 10 random dialogues from the cluster.
Result: 23 topics, including "payment issues," "delivery complaints," "documentation requests." Topic recognition accuracy by the first document – 89%.
Critical nuance: without UMAP dimensionality reduction, clustering metrics drop by 20–30% due to the "curse of dimensionality." We configure n_components=15, metric='cosine' – this increases Silhouette Score from 0.25 to 0.41.
What Is Included in the Work
We offer a full turnkey clustering implementation cycle in 2–3 weeks:
- Corpus audit: assessment of volume, quality, and language characteristics.
- Selection of embeddings and algorithms for your task (we test 3–4 configs).
- Pipeline implementation in Python:
pandas+scikit-learn+umap-learn+hdbscan/bertopic. - Testing on a holdout sample: metrics + visual analysis of the map.
- Deployment as a Docker container or REST API on FastAPI.
- Documentation: retraining instructions, cluster descriptions, dashboard (Grafana/Streamlit).
We guarantee transparency – you receive reproducible scripts and a model description in Model Card format.
Quality Assessment Without Labels
Internal metrics help filter out failed configurations:
- Silhouette Score – compactness and separation of clusters. Target value >0.3.
- Davies-Bouldin Index – the lower, the better.
- Coherence (UMass/Word2Vec) – semantic coherence of cluster top words. For Russian text, we use trained fastText vectors from RusVectōrēs.
Additionally, we perform expert validation: a random sample of 100 documents, labeled by 3 analysts. Disagreements are resolved by voting – this provides ground truth for manual accuracy calculation.
Typical Mistakes in Text Clustering
- Ignoring stop words and lemmatization: embeddings get "clogged" with frequency noise, and clusters lose meaning.
- Too low
min_cluster_sizein HDBSCAN (less than 5): we get hundreds of microclusters that are statistically insignificant. - Direct clustering of 768-dimensional vectors without UMAP: the curse of dimensionality destroys metrics.
- Using only one quality metric: Silhouette is good for K-Means, but for HDBSCAN it is better to combine it with Davies-Bouldin.
Why Work With Us?
Our experience in NLP is 5+ years, over 50 completed text processing projects, including clustering, RAG systems, and fine-tuning LLM. We understand how to train a model on your corpus so that the result is stable in production. Cost is calculated after audit.
To evaluate your corpus and select the optimal pipeline, contact us – we will prepare a turnkey solution in 2–3 weeks. Order a pilot project to see the result on your own data.
