Why crypto news parsing breaks across dozens of sources
A single script that fetches headlines from one source once a minute works for years. Across thirty sources, data collection from sources falls apart: some feeds serve RSS and Atom, some — only JSON via a closed interface, some — ready-made markup without explicit dates.
A source changes its response structure — the parser silently returns an empty output, and this is noticed not by an error in the log, but by an asset disappearing from the feed.
Then time and identifiers diverge. Timestamps arrive in different time zones and formats, identifiers are unique only within a source, and a republication arrives with a different URL and a slightly modified headline.
Without time normalization and fuzzy text matching, a crypto news aggregator shows the same item three times and at the same time loses the primary source.
The main cost of error is missed publications.
During periods of high volatility, publication frequency increases severalfold, while rate limits remain the same: the source responds with a refusal, the polling queue grows, and the window between publication and appearing in the crypto news feed stretches from seconds to minutes. It is precisely during these minutes that the news has weight.
There is also a silent failure: a source serves only the latest entries, and during a burst of more than a hundred publications, older ones are displaced before they can be retrieved. Below we examine how crypto news parsing is built so that the feed does not break during peaks.
What a unified data stream provides: speed, completeness, fault tolerance
A unified data stream is not a single parser, but an ingestion layer where RSS feeds, public exchange APIs, and media outlets such as CoinDesk converge independently.
Each publication receives a normalized representation: source, timestamp, link, and content fingerprint, which is used to remove duplicates — the same news item arriving from three places enters the feed once.
The delay before a news item appears consists of polling frequency and processing time.
Sources with a dense publication flow are polled with priority, while infrequent ones are polled less often so as not to consume API quotas; as a result, a news item appears in the feed tens of seconds after publication, rather than after hours of waiting for manual review.
The completeness of source coverage rests not on the number of connected feeds, but on miss control.
The share of missed publications is measured by comparing the source archive with what reached the feed: discrepancies go into backfill, and typical causes — a changed response schema, redirect, non-standard date format — are addressed with separate parsing rules and an alert when the share of successful parses drops.
Feed fault tolerance is built on isolation: each source is a separate handler with its own state, retries, and increasing pause between attempts.
If one source fails, the others continue to fill the stream, and after recovery the service backfills the missed window by timestamp — no visible feed downtime occurs, and the loss is limited to one polling interval.
Comparison of collection methods: direct connections, ready-made feeds, page parsing
Data collection methods differ not in quality, but in which source limitations you are willing to accept. A direct connection to a source delivers the full stream with minimal delay, but requires keys, quotas, and your own handling of limits for each node.
Ready-made news feeds provide a normalized stream immediately — at the cost of someone else's polling frequency and field composition.
Page parsing of publications covers sources without programmatic access, but it is the most fragile: a layout change breaks the extraction, and protection against automated requests limits request frequency.
In practice, an aggregator combines modes: feeds cover a broad range of publications, direct connections cover the key ones, and parsing covers the remainder.
| Option | When it fits | Limitations |
|---|---|---|
| Direct connection to a source | Full stream, custom filters, and low latency are needed | Access keys, quotas, rate limits, and a different response format for each source |
| Ready-made news feeds | Fast coverage of a broad range of publications is needed | Dependence on someone else's normalization, delivery delay, truncated fields, supplier changing terms |
| Page parsing of publications | The material is on the site but without programmatic access | Fragility to layout changes, protection against automated requests, lower update speed |
The choice of mode is fixed separately for each source and revisited when the provider changes terms or layout.
Stages of parsing implementation: from source audit to handover
The order of work is structured so that each next step relies on a verified previous one: first we establish which sources actually provide usable data, and only then do we build a stream around them.
The output is not a set of one-off scripts, but a maintainable service with predictable failure points and a clear cost of modifications.
- Source audit. We inventory feeds and access points: output completeness, publication frequency, request limits, usage rules. We separately flag duplicate sources and those that serve only headlines.
- Data flow schema. We agree on the record route: ingestion, normalization, deduplication, enrichment, storage, delivery to consumers. We fix the field set, coin matching rules, and system behavior if any node fails.
- Collection setup. A crawl schedule that accounts for time zones and publication peaks, retries with increasing delay, read position persistence — a restart does not lose or duplicate records.
- Cleaning and enrichment. We bring texts to a unified form, strip markup and ad inserts, match projects against a synonym dictionary, and assign topic and sentiment.
- Testing on a live stream. Several days on real sources: we count the duplicate share, news appearance delay, misses, and behavior when a feed is unavailable. We verify the sample against manual review.
- Monitoring. Alerts when the share of successful fetches drops and duplicates rise; if one source fails, the stream continues working on the others.
- Project handover. We hand over the flow schema, source configuration, instructions for connecting a new feed, dashboards, and an analysis of typical failures — plus training for the team that will run the service independently thereafter.
Project artifacts: flow schemas, normalization rules, procedures
After delivery, the client's team has a set that allows them to develop the system without us: it answers three questions — where data comes from, how it is brought to a unified form, and what to do when a source stops serving its feed. Below is the basic set that we hand over in every project.
- Data flow schema — RSS and API sources, queue, normalization and storage layers, delivery points; with notes on where safeguards are placed when a feed is unavailable.
- Source configurations — lists of feeds and endpoints, polling frequency, request limits, timeouts, retries with increasing pause.
- Normalization rules — mapping fields from different feeds into a unified model, converting time to UTC, canonicalizing tickers and categories, behavior with partially filled records.
- News deduplication rules — comparison by canonical URL and headline fingerprint, reconciliation window, text similarity threshold, conflict resolution order between sources.
- Operations procedure — how to disable a noisy feed, connect a new one, replay a batch of records after a failure, whom to notify when the stream lags.
- Observability — a dashboard for data freshness, share of discarded duplicates, and parsing errors, plus alerts when a source is silent longer than a specified interval.
- Instructions and tests — local startup procedure, saved feed responses, checks for broken markup and format changes, so that fixing one parser does not break neighboring ones.
Case study: how an aggregator feed stopped losing publications
The client is the editorial team of an industry media outlet, KriptoVestnik: the feed was assembled from several dozen sources, some delivered data as a stream, and some through public interfaces of exchanges and analytics services.
There was no scheduled polling: sources were connected chaotically, and the feed survived on manual work by an editor.
What it was before
In a flow measurement over two weeks, 8–14% of publications did not reach the feed: duplicates displaced fresh records, and a failure of one source quietly left a blockage in the common queue.
The delay from publication to appearing in the feed stayed within 20–40 minutes, and on days of sharp market movements stretched to an hour.
The collector crashed 3–4 times a week — it did not survive a connection drop and continued returning empty responses until an operator noticed the loss.
What it became after
We rewrote collection: a unified queue with processing acknowledgment, a uniqueness key based on headline and publication time, a separate buffer for re-parsing failed responses, and lag monitoring for each source.
Losses fell below 0.5% — this is noise from materials deleted on the source side. The median feed delay is 40–90 seconds, including parsing and duplicate checking.
Over a quarter of observation there was no downtime: a connection drop leads to tasks being requeued, not to the feed stopping, and recovery takes minutes.
The editorial team stopped manually appending to the feed and returned to working with texts: misses are visible in monitoring for each source, rather than through a reader complaint.
How does RSS collection differ from working with an exchange API?
These are different classes of sources, and confusing them is a typical mistake when designing an aggregator. A feed serves already written materials, while an exchange interface serves primary market data: trades, quotes, volumes over an interval. The choice between them determines data latency and what will end up in storage at all.
RSS collection covers a text stream with a single module: the Atom format and its predecessor are read the same way, hundreds of publications are connected by editing the address list, without keys or agreements.
The limitation is the output window: a feed serves the last 10-50 items and does not store an archive, so the depth of publication history begins at the moment collection starts.
Data latency is from a few minutes to half an hour, for some publications more, and each source's markup has to be parsed separately.
| Option | When it fits | Limitations |
|---|---|---|
| RSS collection | News stream, reviews, and analytics from dozens of publications | 10-50 item window, delay up to half an hour, its own markup for each source |
| Working with an exchange API | Quotes, trades, volumes, listing and delisting events | No publications at all, rate limits, a key is required, and paginated output |
An exchange interface provides what feeds do not: the exact time of an event and the numbers on which metrics and signals are built. In exchange — request quotas, limited depth for historical candles, and dependence on the availability of a particular venue.
Comparison of data sources almost always ends in a hybrid scheme: market indicators come directly, texts through feeds, and the full text of a document is retrieved from the publisher's page.
Technical questions: limits, duplicates, delays, source failures
An aggregator breaks not on parsing a single feed, but on edge cases: a source closes access under frequent requests, the same news item arrives through several channels, a publication appears later than the primary source. Below is how the system behaves in these scenarios.
The source limits request frequency — what then?
We poll not on a rigid schedule, but through an accumulating priority queue: the source's update frequency determines the order, and the "retry later" flag from the response shifts the next request.
If the refusal persists, the channel is switched to a backup acquisition path — a newsletter or a feed mirror.
How do you distinguish a duplicate from a repeat publication?
We compare by three attributes: canonical link, normalized headline, and body text fingerprint. A match by link and fingerprint is a duplicate: we merge it into one card and keep the list of sources.
A match only by headline with edits in the body is a developing story; we update the record rather than create a second one.
A news item arrived with a delay — how do you avoid serving stale content?
We store two times: the moment of publication at the source and the moment of detection on our side. The feed is sorted by the first, and the discrepancy shows how much the channel lags — a systematically lagging one is lowered in polling order. Materials older than a specified window go to the archive and do not enter the output.
The source changed its feed format or is unavailable?
Every response is checked against the expected schema: if fields are missing or renamed, the channel goes into quarantine and does not write garbage to the database, an engineer receives an alert and fixes the parsing schema.
During the fix, the aggregator runs on the remaining channels — the stream degrades but does not stop.
Let's discuss the task: what is needed to evaluate your aggregator
Evaluating an aggregator project relies on inputs, not guesses. The more precisely the flow of materials and the requirements for the result are described, the fewer assumptions will enter the data collection work plan — and the faster we will return with an estimate.
- List of sources: open feeds, programmatic interfaces of exchanges and media, pages without official access. For each, it is important to understand whether the source provides data natively or markup parsing is required.
- Volume and frequency: how many items arrive per day and how quickly they must appear in the database. This determines the polling schedule and load headroom.
- Normalization schema: which fields are needed in the output — headline, source, publication time, tickers, sentiment — and in what form the aggregator delivers them to the consumer.
- Deduplication rules: how to merge the same story from a dozen sources and which one to consider the primary source.
- Storage and history: archive depth, whether snapshots of previous text versions are needed, requirements for search across accumulated data.
- Access and limitations: keys, request limits, source usage terms, mandatory data labeling.
- Post-launch support: who monitors feed interruptions, how quickly a response is needed, and what we consider degradation.
Describe the task — we will return with an estimate and a plan where we separately discuss stages, risks, and post-launch support.