Custom Content Aggregator: RSS, API Parsing, Dedup, ML, Personalization

Collecting content from dozens of sources manually takes hours of copying, duplicate checking, and sorting. An automated content aggregator solves this: it parses RSS and APIs, filters duplicates, categorizes, and presents a personalized feed. We design and implement such a pipeline for your project — from architecture to deployment. Over the years we have launched more than 10 aggregators for media and corporate portals, reducing manual content-gathering by 80%. Budget savings reach 60% thanks to automation.

A typical project includes 30–50 RSS feeds, 5–10 API sources, and several websites for scraping. Each source requires a separate adapter with rate-limit and error handling. We use Bull job queues on Redis, allowing parallel processing of up to 100 sources without data loss. On failure, a task automatically retries with exponential backoff up to 3 times. Monitoring via Grafana tracks the success rate of each source.

Pipeline Schema

Why Deduplication Is Key

Duplicates appear when the same story is published by multiple sources. We apply three methods:

SimHash is 3× more effective than exact hashing for detecting similar texts, reducing false positives to 5%. With proper tuning, deduplication filters out 95% of duplicates. For one media project with 50 RSS and 10 API sources, we set up a pipeline that processes 500 articles daily, slashing manual work from 4 hours to 15 minutes. More on SimHash.

from simhash import Simhash def is_duplicate(text1: str, text2: str, threshold: int = 5) -> bool: h1, h2 = Simhash(text1.split()), Simhash(text2.split()) return h1.distance(h2) < threshold 

What Parsing Tools Do We Use?

RSS parsing is straightforward. The challenge is extracting clean text when scraping:

• Readability (Mozilla) – strips navigation and ads;
• Trafilatura (Python) – extracts text with language detection;
• Playwright – for SPAs requiring full JavaScript rendering.

Each adapter takes 1–2 days to set up, including error handling and rate limits. Parsing speed reaches 10 articles per second per adapter. Readability docs: Readability.

Categorization and Tagging

Automatic classification by topic:

• Keyword matching: rules like "rouble" → "Finance";
• ML classification: fastText or BERT-based for multi-label tagging. ML accuracy reaches 90% on labeled data. Processing 1000 articles per minute is a realistic speed for fastText.

Language detection: langdetect (Python) or franc (Node.js).

How Feed Personalization Works

The user manages filters:

• enabled/disabled sources and categories;
• keyword subscriptions;
• negative keywords to exclude topics.

Optionally, algorithmic ranking: collaborative filtering based on reading history of similar users. This boosts engagement by 30% according to our data.

Copyright Compliance

The aggregator shows only previews (lead + link), respects robots.txt and rate limits. The fair use model allows snippets but not full republication. We always credit the source and author.

What Is Included in the Work

• Source analysis and architecture design;
• Pipeline implementation: parsing → normalization → deduplication → storage → delivery;
• Indexing setup in Elasticsearch/Meilisearch;
• Categorization (rules or ML);
• Personalization (filters and ranking);
• API and admin panel documentation;
• Testing and load testing;
• Deployment with monitoring (Grafana, Sentry).

Timeline

Cost is determined individually after an audit. Order an audit of your sources — we’ll evaluate in one day. We guarantee deadlines and confidentiality. Our team has launched over 10 aggregators. Contact us to discuss your project.