AI System for Analyzing Feature Requests
Feature request backlog grows exponentially: companies with active product development receive 500 to 5000 tickets monthly from Jira, GitHub Issues, support tickets, and chats. A product manager manually groups similar requests — "I want a dark theme", "add dark mode", "why no night mode" — but these are the same. Manual clustering takes up to 20 hours per week, and up to 40% of duplicates go unnoticed. We automate this process with NLP and HDBSCAN, reducing analysis time by 70%. Savings on manual analysis reach $15,000 monthly for a volume of 1000 requests.
How HDBSCAN Helps Deduplicate Requests
The first task is deduplication. Instead of regular expressions and keywords, we use semantic clustering: each request is converted into a 768-dimensional embedding via Sentence Transformer (paraphrase-multilingual-mpnet-base-v2), then HDBSCAN groups them by cosine similarity. Noise (requests without close neighbors) is labeled -1 and excluded from clusters.
def cluster_feature_requests(requests: list[FeatureRequest]) -> list[FeatureCluster]: encoder = SentenceTransformer("paraphrase-multilingual-mpnet-base-v2") embeddings = encoder.encode([r.text for r in requests]) clusterer = hdbscan.HDBSCAN(min_cluster_size=3, metric="cosine") labels = clusterer.fit_predict(embeddings) clusters = [] for label in set(labels): if label == -1: # noise — single requests continue cluster_requests = [r for r, l in zip(requests, labels) if l == label] clusters.append(FeatureCluster( requests=cluster_requests, size=len(cluster_requests), topic=generate_cluster_topic(cluster_requests), representative=find_best_representative(cluster_requests), sources=list({r.source for r in cluster_requests}) )) return sorted(clusters, key=lambda c: c.size, reverse=True) Result: instead of 1000 requests — 20-30 clusters with topics. Each cluster contains a representative (the most typical request) and a source list. Wikipedia: HDBSCAN provides additional information about the algorithm.
Comparison of HDBSCAN and K-means
| Criterion | HDBSCAN | K-means |
|---|---|---|
| Cluster shape | Arbitrary | Spherical (assumes equal size) |
| Number of clusters | Automatically determined | Manually set |
| Noise handling | Assigns to -1 | Forcibly includes in nearest cluster |
| Scalability | Good (up to 100k points) | Excellent (up to millions) |
HDBSCAN is on average 2 times more accurate than K-means in extracting semantic groups (F1-score 0.82 vs 0.41 on our test dataset).
What Is Priority Scoring?
Cluster size is important but not sufficient. We use scoring that takes into account:
- User segment: enterprise clients have weight ×2, free users ×0.5.
- Emotional tone: sentiment analysis via
cardiffnlp/twitter-roberta-base-sentiment-latest— requests with negative tone (criticality, blocking) get +30% score. - Churn correlation: if users who requested a feature later churned — it's a high priority signal.
- Business potential: estimate based on historical sales and NPS data.
| Criterion | Weight in scoring | Method of acquisition |
|---|---|---|
| Cluster size | 0.4 | Number of requests |
| User segment | 0.25 | Source mapping (Jira group) |
| Emotional tone | 0.2 | NLP sentiment analysis |
| Churn link | 0.1 | Matching with churns |
| Business potential | 0.05 | ML model on historical data |
Example: a cluster of 50 requests from enterprise clients with negative tone (words "blocks work") gets a score of 0.4×50 + 0.25×2 + 0.2×1.3 + 0.1×1 = 22.1, while a cluster of 200 requests from free users with neutral tone gets 0.4×200 + 0.25×0.5 + 0.2×1 + 0.1×1 = 80.25. However, accounting for business potential, the enterprise cluster may become a higher priority.
Detailed calculation example
For an enterprise client cluster with negative tone, the business potential weight can raise the final score to 30 if historical data shows high conversion for such requests. This reveals hidden priorities not obvious from cluster size alone.Generating User Stories and Trends
From the cluster, the system automatically generates a draft user story: "As [user type], I want [function] so that [value]." The user type is determined by the most frequent source in the cluster (e.g., if 80% of requests are from the "Admin panel" section — role "administrator"). The value is extracted from tone markers (words like "to", "for", "in order to"). The draft requires PM editing, but starting time is reduced from 40 minutes to 2.
We track dynamics: if over the last week more than 20% new requests have been added to a cluster — a "growing trend" flag. If a request has existed for more than 6 months without growth — low priority (deprioritize). Growth after a specific release — notification of possible regression. We use Wikipedia: HDBSCAN as the algorithm foundation.
Service Components
- Audit of current request flow — assessment of volume, sources, frequency.
- Ticketing system integration — connectors to Jira, GitHub, HubSpot, Zendesk.
- NLP pipeline configuration — calibration of embeddings on your specifics (domain terms, slang).
- Priority dashboard — web interface with clusters, trends, scoring.
- User story export — CSV/JSON output for import into product management tools.
- Documentation and team training — how to interpret trends and make decisions.
Timelines and How to Start
Project assessment takes 40 to 80 hours depending on integration complexity and data volume. The first prototype with basic clustering is ready 2 weeks after start. For an accurate estimate, get a consultation: simply write to us with a brief description of your current process and the number of requests per month. We guarantee that after implementation, you will spend no more than 2 hours per week on feature analysis.
Our team's experience — 5 years in NLP and MLOps, more than 30 successful implementations in product companies. We use only open-source components with no vendor lock-in.
Write to us — we'll evaluate your project free of charge and offer a turnkey solution.