Content aggregator development: where data collection breaks
Content aggregator development starts not with the interface but with collecting data from sources: dozens of platforms return cards with different structures, and most projects break at the junction of ingestion and normalization.
While there are five sources, the scheme works with any logic; at the fiftieth, the catalog begins to diverge from reality, and this can no longer be fixed with settings.
Sources rarely report an error — instead of a failure, a response with empty fields arrives, and the aggregator writes a card without a description or a key characteristic.
Such an entry looks valid, enters the index, and goes to the user, and is discovered only through a complaint. Alongside this are source limits: a crawl schedule designed for even load hits a daily quota, and some platforms silently drop out of the cycle until the next window.
Card duplicates spread unnoticed: the same object comes from a supplier and from a distributor, the names differ by a function word and word order, and the matching key does not trigger.
Three similar records with different characteristics appear in the catalog, search loses relevance, and manual cleanup can no longer keep up with the flow.
Stale data in the catalog accumulates even faster: a full crawl cannot cover all platforms in time, updates lag by a day, and the card state diverges from the source.
Below we break down how this is closed — from crawl prioritization and a retry queue to duplicate merging and field completeness control.
What a managed aggregator delivers: effect in measurable values
The overview is based on engineering values that are measured on a running catalog: publication delay, duplicate share, resilience to source failures, card fill rate.
Catalog freshness and update frequency
Source crawling runs on a schedule with different frequencies for different feed types: where records appear every minute, the interval is short; where updates are rare, it is long.
The gap between publication at the source and appearance in the catalog is reduced to minutes instead of hours. Incremental crawling by change indicators updates only new and edited items, while a full pass remains a safety net once a day.
Reducing the duplicate share
Comparison happens on two levels: exact match by normalized text and source identifiers, then fuzzy — by signatures of the title and body.
The duplicate share in results drops from tens of percent to single digits, and republished materials are collected into one card with a link to the original source.
Tolerance to source failures
Changed markup, request rate limiting, unavailability for several hours — each case has its own response: retries with increasing delay, saved last successful responses, a fallback parsing method.
The catalog continues to be filled from other sources, and a returning source is caught up without losses — the class of failures where "output stopped entirely because of one site" goes away.
Relevance of results and user selections
Cards are enriched with features: topic labels, entities, view and click behavior. Ranking relies on their combination, so the share of relevant cards in the top lines grows, and completeness — the fill rate of required fields — stays above 90% due to intake checks.
Content aggregator development: collection and update modes
The collection mode determines two things at once: how many minutes it takes for new material to enter the catalog and what share of requests to sources we waste.
A full export once a day is predictable, but on sources with a dense publication flow, almost all responses return records that are already known.
Scheduled source crawling
Incremental crawling requests only what appeared after the last synchronization — by timestamp and material checksum. An overlap window covers edits at interval boundaries, and the cursor for each source is stored separately: a failure on one resource does not shift the others. Freshness — from tens of minutes, sample volume — several times smaller.
Event-driven updates via source signal
The source itself reports a publication — via a change feed or a changed sitemap. Delay drops to minutes, but a failure class appears: a missed signal. Therefore event-driven mode is supplemented with a daily reconciliation; otherwise the catalog quietly diverges from the source.
| Option | When it fits | Limitations |
|---|---|---|
| Full export | One-time reconciliations, recovery after failure | Hours per cycle, many already known records |
| Incremental crawling | Regular flow with publication date | Needs a cursor and overlap window |
| Event-driven updates | Fast feeds, exchanges, marketplaces | Depends on signal, safety reconciliation needed |
| Crawling via job queue | Many sources with different frequencies and limits | Priorities, retries, rate limiting |
| Manual upload of slices | Closed partners, one-time sets | Does not scale, freshness depends on a person |
In practice, modes are combined: a job queue distributes priorities, incremental crawling works by default, and event-driven mode is enabled where a minute matters.
According to industry measurements Ahrefs, a significant share of pages is not updated for years — such sources are served well enough by rare crawling, and the request budget goes where changes are real.
How implementation proceeds: stages of engineering work
Before the start, we capture what already exists: sources, their formats, request rate limits, volume and terms of content use. Below is the order of work, which shows what step we are at and what the client can verify.
What we do at the source analysis stage
- Source analysis: we inventory feeds, partner exports, and file exports; note request limits, stability of record identifiers, and terms of content use.
- Data exchange scheme and entity model: we describe how a record arrives, is normalized, and is linked to the author, category, and media, so that duplicates and updates land in one card.
- Crawl setup and merge rules: schedule and crawl depth, source priority in field conflicts, text similarity thresholds, and record version merge logic.
- Data quality checks on slices: we run historical samples, count the share of found duplicates, field completeness, and the number of false merges.
How we verify the result before handover
- Monitoring and alerts: metrics per source — error share, record appearance delay, empty samples; alerts when a source goes down or the response structure changes.
- Project handover: access, configurations, procedure for connecting a new source, incident review with support, and a support period.
Project artifacts: schemes, configurations, procedures, and access
A project is closed not with a presentation but with a set of artifacts by which the system can be operated and developed without us. Each document is tied to an action: what to run, where to look, to whom to grant access, what to do in case of failure.
Documentation and instructions
- Collector and schedule configurations — source crawl frequency, pauses between requests, retries on errors, queue priorities. They are changed by editing a file, without rebuilding the service.
- Data flow scheme — the path from source to card in results: where normalization is, where enrichment is, where manual queue moderation is.
- Deduplication and entity matching rules — which features are compared, what match share counts as a duplicate, how similar records are separated.
- Update and failure response procedure — release frequency, rollback procedure, notification thresholds, and on-call actions.
- Support instructions — typical incidents and first steps: source not responding, queue grew, duplicates reached the results.
Rights and access separation
- Access matrix and roles — who edits the source list, who changes deduplication rules, who has access to the production environment.
- Procedure for granting and revoking rights — request, confirmation, logging; when an employee leaves, access is closed with one action.
- Change log — which configuration edit changed what in the results and when, so that discrepancy analysis takes minutes, not days.
The set remains with you: the support team is onboarded into the project using these documents, without word-of-mouth retelling.
Case study: how duplicates and catalog update time were reduced
The project is a content aggregator: dozens of sources, hundreds of thousands of cards, daily reindexing. Measurements were taken from the collection service and job queue logs before the rework and one month after — the source data is in the load report.
What it was like before the collection rework
Each source was polled on its own schedule, a full catalog crawl took about six hours, and a card appeared in search with a delay of up to a day.
Deduplication looked for an exact match of page address and title, so 28–34% of results were duplicates: the same product came from several sources and lived as separate cards.
No one controlled pagination crawling: when markup changed, the source silently returned the first page, and up to 11% of positions did not enter the index.
An operator merged 150–200 cards per shift manually, edits were not recorded anywhere, and discrepancies surfaced only in the results.
What was changed: collection was moved to a single pipeline — field normalization, comparison by feature vector instead of exact string matching, automatic merging, and manual review only for disputed pairs.
Reconciliation and search are kept on the current Elasticsearch branch, and a markup validator is responsible for the source scheme.
After the rework, the duplicate share is 3–4%, and these are genuinely different configurations. A full catalog update takes minutes instead of hours, and the delay for a position to appear has been reduced from a day to tens of minutes.
Gaps dropped to fractions of a percent: when markup changes, a notification arrives instead of silent degradation. Manual edits are down to a few dozen per week — only edge cases that the pipeline flags itself.
How scheduled crawling differs from event-driven updates
Scheduled crawling and event-driven updates solve the same task — deliver a fresh version of a position to the catalog — but they pay for it with different resources. The first option polls the source at equal intervals and requires nothing from it except a stable response. The second — the source itself reports a change, and the position is updated almost immediately.
The main difference is the delay in position appearance. With a schedule, it equals half the interval on average and the full interval in the worst case: a product changed a minute after the pass, but the catalog learns about it only on the next one.
The event-driven scheme reduces the delay to the signal processing time, but it runs into the source's readiness to provide such signals: some suppliers provide only batch exports, and event-driven mode cannot be built there.
The second difference is load on the source. A schedule produces a predictable peak: tasks start synchronously, the source receives a burst of requests, which is spread out with windows and concurrency limits.
Event-driven updates distribute load over time but make it uneven: during a mass recalculation of a supplier's catalog, signals arrive in batches, and they must be accepted with queue capacity headroom.
The third is maintenance. A schedule fails silently: a missed pass leaves the old version in the catalog, and this is noticeable only by the freshness metric, so lag control and cycle failure alerts are needed.
The event-driven scheme requires idempotent processing of repeated deliveries, deduplication by content hash, and a reconciliation crawl: without it, one lost signal leaves a position stale indefinitely.
The choice is transparent: a schedule is sufficient where lateness is cheaper than infrastructure — prices, stock, availability. Event-driven updates are needed when a position must match the source within minutes and the source is ready to signal.
A working scheme is usually hybrid: signals provide freshness, and a reconciliation crawl provides the completeness guarantee.
Technical concerns: load, duplicates, legality of collection
We build source crawling so that the load on them does not exceed normal user traffic: request rate limit per domain, pauses between series, daily ceiling per source.
If a source fails, it leaves rotation, unprocessed addresses return to the queue, and the catalog does not lose positions.
What happens when source markup changes?
The collector saves the raw response and parses it in a separate layer. If the layout changes, parsing fails, not loading: the share of unsuccessful cards is visible, the rule is fixed, and what has already been collected remains in the catalog.
Will the catalog be preserved if crawling is stopped?
Yes. Writing happens in batches, and the queue state and cursors are recorded. After restart, crawling continues from the stopping point, without a repeated pass over the source; duplicates are cut off by deduplication by key and content fingerprint.
How is a new source connected?
By a separate description: addresses, parsing scheme, crawl frequency, deduplication rules. The source is included in the common flow without affecting working ones, and you can start with one and add the rest gradually.
Is data collection lawful?
We work with open data, checking against the Directive on the Legal Protection of Databases: we take into account the source's terms of use, limit sample volume, and store a link to the original source and a timestamp. We do not collect personal data without a legal basis and do not mix it with the catalog.
Let's discuss the task: what is needed to come back with an estimate and plan
To make the project estimate substantive rather than a "rough range," we need input from you — and this is not a formality: at the input we count the number of sources, deduplication volume, and load.
Below is what is useful to send and what to agree on before the start, so that we come back with a work plan by stages.
- List of sources and their types: APIs, feeds, pages for parsing, file exports. For each one, it is important to understand whether it provides structured data or only markup — the labor intensity depends on this.
- Access to sources or test slices: keys, limits, allowed request rates. If there is no access, an export for several days is enough — on real data we immediately check how deduplication works.
- Current catalog scheme: entities, rules for linking one object from several sources, required fields, and what has already been accumulated. This defines the scope of work on merging and storage.
- Requirements for update frequency and merge rules: how quickly a new object must enter the catalog, which source has priority in field conflicts, what to do with disappeared offers.
- Infrastructure constraints: deploy with you or with us, requirements for logs and personal data, whether external services are prohibited.
- Who supports after handover: your team or ours, whether a source failure response procedure is needed, access to monitoring, and training.
- Expected load and personalization: number of users, features for recommendations, whether there is behavior history.
Describe the task in free form — we will come back with an estimate and a work plan broken down by stages with timelines.
