Setting Up Crypto Data Parsing: Where Collection Breaks
Scheduled crypto data collection looks like the simplest part of the system: a job wakes on a timer, pulls the order book and trade feed, and stores them.
It breaks not on a single run, but over weeks of continuous operation — lag accumulates, and it is no longer visible from the logs of an individual pass.
The first break is gaps in data. The run schedule starts a job every few seconds, and if the previous pass did not finish, the scheduler either stacks duplicate copies or shifts the window.
A hole in the series is not discovered immediately: some candles are simply missing, and calculating returns and volatility on a hole-riddled series produces a biased result that looks plausible and flows further into reports.
The second is discrepancies between sources. Exchange data parsing runs into the fact that venues return different things: one gives the last trade price, another the interval-weighted average, a third splits the stream into batches at different intervals.
Source timestamps also do not match, and stitching series together without an explicit alignment rule turns analytics into a set of random numbers.
The third, and most expensive, is silent job failure. The source changes its response format or starts returning an empty list, the job finishes with a success flag, and alerts stay silent. The missed window is discovered a day later, when it can no longer be restored by re-requesting.
Below we break down how this is covered: idempotent passes, window completeness control, and source reconciliation on every cycle.
What Regular Exchange Data Collection Provides
Regular collection turns a set of scattered snapshots into a continuous time series: one instrument, one interval, one recording schema.
Market data completeness is measured not by the number of successful requests, but by the fact that a gap is visible — if a source did not respond in its window, an explicit gap marker remains in the series, not a shift of neighboring points.
Without this, a missed hour is quietly glued to neighboring ones, and analytics calculates a price movement that did not exist in the market.
Update latency is set in advance and maintained: streaming subscriptions give an even lag for the order book and trades, scheduled polling gives a uniform step for closing candles.
Floating lag is more dangerous than a large one — a strategy tested on data collected “as it happened” shows a result that does not repeat in live execution.
Collection continuity is a separate reliability loop. The scheduler restarts failed jobs, retries use increasing delay, and duplicates are suppressed by the key “source + instrument + timestamp”: a connection drop or restart does not corrupt the series or double volumes.
Quote history is an asset that cannot be restored retroactively: missed days cannot be backfilled from any source.
Therefore we verify series integrity by reconciling with an independent aggregator, such as CoinGecko, and keep a coverage report for each instrument — the share of actual points versus expected points for the period.
Data Acquisition Modes: Comparison of Approaches
The collection mode determines how well the data keeps up with the market and how often discrepancies have to be investigated.
Run schedules, a persistent channel, ready-made vendor exports, and manual requests solve one problem with different means — they differ in latency, completeness, and resilience to source limitations.
The choice is dictated by the sources themselves: public nodes return history, but throttle request rates and cut off access for exceeding limits; paid data providers give an even feed, but fix the set of fields and request quotas. Below are four modes and where each one breaks.
| Collection mode | When it fits | Limitations |
|---|---|---|
| Timer-based run schedule | Regular snapshots: quotes, pools, wallet balances, history accumulation | Latency equals the interval; a missed run leaves a gap in the data |
| Persistent channel (event subscription) | Trades, mempool — where second-level reaction matters | A connection drop tears the stream: reconnection, duplicate filtering, and a buffer are needed |
| Ready-made vendor exports | History and aggregates: candles, address metrics, labeled events | The set of fields is fixed; publication latency; dependence on a single provider |
| Manual on-demand requests | One-off checks, incident investigation, reconciliation of disputed transactions | No history or reproducibility; does not scale to a stream |
In practice, modes are combined: the schedule accumulates history, the persistent channel takes hot data, and vendor exports serve as reconciliation. Such a set covers both connection drops and source rate limits.
Stages of Setting Up Regular Data Collection
Regular crypto data collection breaks not on an individual request, but on the schedule: the source changes its response format, some points arrive late, duplicates appear in storage, and gaps are noticed only in the report.
The workflow is structured so that the collection schema is fixed before launch, rather than reconstructed after a failure.
- Source audit. We check exchanges, aggregators, and network nodes: which fields are returned, with what latency, whether there are request rate limits, and whether history for past periods can be exported.
- Collection schema. We fix a unified recording format, field composition, timestamps, and polling mode — scheduled or on new block arrival. The schema becomes a contract: discrepancies are caught at input, not in the finished report.
- Request retry rules. We describe the number of attempts, the pause between them, and the criterion by which we distinguish a temporary failure from a changed response; after attempts are exhausted, the source is marked as problematic and triggers an alert.
- Test run. We run collection on historical data: compare record volume, completeness for each source, and match against already accumulated values for the same period.
- Acceptance. We compare control metrics before and after, check behavior when a source is unavailable, and confirm correct continuation of collection after a forced stop.
- Handover of procedures. We hand over the schedule, the procedure for connecting a new source, and troubleshooting of typical errors — support no longer depends on a specific contractor.
What Is Delivered: Schemas, Configurations, Procedures
In addition to a working collector, we hand over a set of documentation and configurations that your team can use to support and extend the jobs without our involvement. Below is what exactly remains in the repository by the time of delivery.
- Data schema — description of entities: candles, trades, order book, funding rates; field types, rounding precision, timezone, and deduplication key (instrument + timestamp + candle type).
- Collector configuration — list of jobs, run interval, query depth, source request limits, pause on failures, list of exchanges and trading pairs in one file.
- Access rules — read-only keys, storage of secrets outside code, rotation procedure, role separation for production and test environments, who can see raw data.
- Restart procedure — step-by-step order after a failure: state check, backfill of the missed window, rerun without duplicates, stop criteria.
- Collection log — job statuses, number of records written per window, source errors, latency from timestamp to write, gap alerts.
- Deployment scheme — where jobs run, how raw data and archives are stored, what retention periods are set by layer.
- Extension guide — how to add a new instrument or field without modifying the core: what to change in the configuration, how to verify the result in the test environment.
The package is designed for an engineer who did not participate in development: from the documentation, collection can be brought up from scratch, and a data discrepancy can be investigated using the log, without contacting us.
Case Study: Restored Data Collection Completeness Across 14 Sources
The client collected market data from 14 sources: exchange APIs, aggregators such as CoinGecko, and public network nodes.
Scheduled runs executed, but 12% of windows were lost: the source returned a rate-limit error, the job failed, and data for that interval did not reach the next cycle. Discrepancies between channels reached 7% — position reconciliation and reporting broke on them.
What we changed: we split collection into three layers — a window scheduler, a pool of loaders with retries and quarantine for “silent” sources, and separate history backfill for missed intervals.
Limits and keys were separated per source, and responses were stored in an immutable log: you can see which window is fully closed, which is partial, and which needs to be re-fetched.
What became: data completeness grew from 88% to 99.7%, discrepancies between channels — from 7% to 0.4%. Recovery time after a source failure decreased from 40 minutes of manual reload to 90 seconds of automatic backfill — the next cycle starts with complete data.
We separately configured alerts: if a window did not close within the allotted time, a signal arrives indicating the source and interval, rather than “something failed”.
Metrics were captured before and after on the same set of sources, so the effect holds on a continuous stream, not on a one-off export.
How Scheduled Parsing Differs from Real-Time Streaming
Streaming gives latency in seconds: a subscription to trades and the order book brings events as soon as they appear on the network.
The price is a persistent connection that must be maintained and monitored: a break of several minutes leaves a hole in history, and it is noticed only during reconciliation.
A schedule works differently. The scheduler starts a job once per interval; it fetches a window of data on request, writes it in a batch, and exits.
Latency is measured in minutes, but there is a single point of failure: if a run did not succeed, the next one backfills the missed interval, and no state is stored between runs.
Support load differs radically. Streaming collection needs an on-call engineer: alerts about reconnections, buffer growth, and stuck subscriptions arrive at any time of day.
Scheduled jobs fail predictably, are visible in the run log, and recovery comes down to rerunning for the required dates.
The downside is that a schedule cannot build event-driven reactions — liquidations or sharp volatility spikes. Streaming, in turn, is poorly suited for exporting months of history: it is expensive in the number of connections and does not allow data to be replayed retroactively.
The practical choice is a hybrid. The schedule maintains history and reference data and backfills gaps; streaming serves metrics where a second matters. Task separation removes both the on-call burden and the risk of data holes.
Technical Concerns: Gaps, Limits, Duplicate Records
Each of these failures is reproducible, and each has an engineering answer — we fix it before launch, not after the first discrepancy.
Below is a breakdown of typical concerns that we resolve during collection design: incomplete windows, venue limits, duplicates, and changes in response format.
The source returned an incomplete window for a past interval — is the data lost?
The window is re-requested with a time margin, not strictly according to schedule boundaries. Each sample is written with a coverage marker: if the response was partial, the interval is marked as not closed and is loaded further by the next pass.
The gap remains in the log — it is visible and can be closed manually, instead of drawing conclusions from an incomplete series.
How are duplicate records excluded?
Each record has a composite key: operation identifier, venue, and point in time. The record is inserted with an update on match — rerunning the same window does not double the volume.
Overlapping windows on the schedule are therefore safe: collection can be restarted as many times as needed to close a gap.
What should be done about source request rate limits?
Polling is distributed over time, rather than going out in one batch. When a limit triggers, the pause increases, the queue is preserved and replayed after access is restored.
We prioritize critical series — secondary sources lag in time but do not bring down the entire collection or produce garbage values.
The source changed its response format — what then?
Ingestion follows an explicitly described response schema, and each record is validated for compliance before being written to storage.
On a mismatch, the data goes to quarantine and an alert is sent to the on-call engineer — collection is marked as stopped for this source, rather than filling the database with values that would later have to be cleaned out.
Let’s Discuss the Task: What to Clarify Before Starting Work
To come back with an estimate and work plan rather than general wording, we need to understand the domain: which assets and venues are included in the sample and what exactly is counted in the stream — trades, order book states, exchange rates, network events.
This determines the storage schema, the set of checks, and how often it makes sense to poll each source. Below is a list of inputs that covers most questions at the start.
- Domain: which pairs, networks, and venues are in focus, whether derivatives and OTC segments are needed or only spot.
- List of sources: exchange APIs, aggregators, network nodes, proprietary price sources — with a note on which provide history and which only the current state.
- Completeness requirements: acceptable share of gaps, what to do with discrepancies between venues, whether retroactive reprocessing is needed and over what horizon.
- Depth and frequency: initial load period and polling schedule — minute, hourly, or event-driven.
- Output format: raw responses, normalized schema, or interval aggregates; where we store the data.
- Access and limits: keys, request-count limits, requirements for network access and node placement.
- Failure procedure: to whom and how we report source downtime, how long we wait for recovery, and by which metrics we consider the data reliable.
Send us these inputs — we will come back with an estimate and work plan, where the stages are broken down by sources and run queues.