Trade Data Collection from Crypto Exchanges

Collecting trades from exchanges: where the trade stream breaks

Collecting trades sounds like “connect to the API and write to the database.” In practice, it is two independent sources: a real-time trade stream and historical backfill for a period.

The first provides low latency, the second provides completeness, and discrepancies arise precisely at their junction.

The most common failure is a connection break with the exchange. While the channel is being restored, trades do not arrive: exchanges have a window in which the stream is not buffered, so after reconnection, history is backfilled by trade IDs rather than by time.

If the backfill is done “with a margin” by hours, duplicate trades appear in the database; if without margin, trade gaps appear. Control by the number of the last accepted record, as, for example, on Binance, closes both issues, but requires storing state for each stream.

Data mart desynchronization manifests later and more quietly.

The analytics mart, reporting mart, and notification cache update at different intervals and produce different volumes over the same interval — a discrepancy with the venue’s reporting surfaces during reconciliation, when restoring what was missed is already expensive.

We examine how this is closed: integrity control at ingestion, continuous reconciliation of data marts with each other, and regular validation against exchange reporting data.

What well-established trade ingestion provides: completeness, speed, reconciliation

Trade ingestion looks like a simple task: subscribe to a stream and write to a table. It breaks on reconnections — trades are lost during the connection break window, and re-subscribing brings duplicates and distorts volumes.

Therefore, the stream is designed as a recoverable journal, not as “recording whatever managed to arrive.”

Reconciliation with reporting runs in a separate loop: aggregates by instrument and hour are compared with venue exports — Binance and other exchanges provide open data in a comparable form.

A discrepancy above the threshold automatically creates a task to restore gaps for a specific window, rather than a full history reload.

As a result, an entire class of failures disappears — the “hole in history.” Manual backfills are replaced by a process, and market data completeness is confirmed by reconciliation figures, not by trusting the application log.

Methods of collecting trades: streaming subscriptions and batch exports

Both modes solve different tasks, and trying to cover both with one of them almost always hits limits. A streaming subscription delivers trades as they appear — it is used for online monitoring, quote calculation, risk control, and arbitrage scenarios.

Batch history export is needed where a complete series for past periods matters: analytics, strategy backtesting on historical data, and reconciliation of reporting with exchange data.

The limitations of the modes are also different. In streaming, the key is to maintain the connection and not lose events on a break: automatic reconnection and backfill from the last confirmed trade ID are needed.

In batch mode, everything rests on request limits and the depth of historical data: the venue provides a window back, and beyond that — only archives.

In practice, the modes are combined: the archive covers the past, the stream covers the present, and at the junction, deduplication and continuity checks by trade ID are put in place. Then the dataset remains complete and without holes after service restarts.

From source audit to trade pipeline handover

Collecting trades from exchanges starts not with code, but with analyzing sources. The work order is fixed: venue audit, pipeline design, setup, historical validation, acceptance.

This sequence saves weeks — a correction at the design stage is cheaper than rebuilding an already accumulated archive.

1. Data source audit. We analyze what the venue provides: a real-time trade stream, archive exports, an API with request rate limits. We record the depth of available history, channel stability, and venue rules for further use of this data.
2. Pipeline design. We design a unified trade model: normalized instrument name, side, price, volume, timestamp in a single time zone, source marker. Here, it is also determined how different spellings of the same instrument are reduced to a single record.
3. Setup of collection and storage. Stream subscription plus history loading, deduplication, a buffer in case of connection loss, and retries. Writing to storage with partitioning by date — selecting the required period does not require a full scan.
4. Validation on historical data. We run the pipeline on a restored segment and compare it with a reference sample: number of trades, prices, volumes, share of gaps and duplicates, delay between a trade and its appearance on our side.
5. Control and solution acceptance. We set up metrics and alerts for stream stoppage, latency growth, and volume discrepancies. Acceptance follows agreed scenarios: access, storage schema, restart process.
6. Handover and training. We show the team how to connect a new venue or instrument, how to read metrics, and what to check when the stream breaks.

Handover package: configurations, diagrams, procedures

We hand over not “server access and an engineer’s contacts,” but a package by which your team can rebuild collection or take it over for support without us. Below is what exactly remains with the customer and why each element is needed in operation.

• Source matrix. For each exchange: which pairs and channels are enabled, streaming subscription or periodic polling, historical load depth, request rate limits, and headroom to them.
• Data flow diagram. The trade path from subscription to write: where normalization happens, where the buffer for connection loss is, where the failure point is, and how re-export of the missed interval is triggered.
• Data model. Trade field dictionary — ID, exchange time and ingestion time, price, volume, side — plus deduplication and idempotent write rules so that a repeated pass does not double volumes.
• Data access separation. Roles of services and people, separate read and write accounts, exchange keys in a secret store, access log to the trade archive.
• Environment configurations. Descriptions of the staging and production environments, retries on failures, volume cutoffs, and retention rules for “hot” and archived data.
• Solution documentation. Map of services and dependencies, procedures for typical failures, recovery from archive, escalation contact points.
• Support procedure. What is checked daily — ingestion lag, interval gaps, volume discrepancies against the reference sample — who responds, within what timeframes, and when to switch to a backup source.
• Acceptance materials. Checksums for the reference period, reconciliation of the sample with the exchange export, and a validation protocol by which acceptance can be repeated.

We hand over the package together with a walkthrough: we go through the data flow diagram and procedure with your engineers and answer questions about configurations. Describe the task — we will return with an estimate and handover plan.

Case study: eliminated trade gaps and data mart desynchronization

Trade collection (trades) on Kraken ran through a streaming channel, and every connection break left a hole in history: after reconnection, only fresh records were pulled in, while the middle of the interval disappeared.

Data marts diverged from the venue by tens of minutes, volumes and trade counts in reports did not reconcile, and the on-duty engineer manually backfilled missed intervals from exports.

We rebuilt the collection loop: the stream is written to a buffer with order preserved, each trade receives the exchange ID and venue timestamp, and writing is idempotent — repeated delivery does not create duplicates.

In parallel, reconciliation with the venue’s historical slices by interval runs, and a discrepancy automatically queues the interval for repeated backfill.

Metrics before and after

• Share of lost trades: 0.3–0.5% of records → zero discrepancies in control reconciliations.
• Time for data marts to converge with the venue: 40–60 minutes → 2–3 minutes.
• Share of manual backfills: 12–15% of intervals → less than 1%, and only at the analyst’s initiative.
• Reconciliation by trade count and volume: discrepancy within 0.01% over a daily window.

After that, the loop runs in support mode: scheduled control reconciliations, alerts when deviation crosses the threshold, and parser updates when the venue’s response format changes.

The customer receives a trade history that can be relied on in calculations, without manual backfills and “trusting by eye.”

What to do if your own export already works but the data does not reconcile?

The discrepancy is almost never in the collection itself, but in what is considered the source of truth.

Your own data export was written for one task — a report, a signal, validation of a trading idea — and imperceptibly accumulated assumptions: local time instead of exchange time, volume rounding, discarding partially filled orders.

While there is one consumer, this is not visible; as soon as there are two, the numbers stop reconciling, and the dispute is not about the data, but about whose calculation is correct.

An audit of the current solution starts with reconciliation, not with reading code. We take a reference interval, match records by the key “symbol + trade time + ID,” and analyze where rows are extra, where they are missing, and where they are time-shifted.

Then we look at logs: connection breaks, repeats after reconnection, lack of deduplication at the window boundary. In practice, two or three such causes explain the bulk of discrepancies.

Next is a parallel run. The new loop writes to a separate schema and changes nothing in the running process: both streams run simultaneously, are reconciled daily, and discrepancies are categorized — window loss, duplicate, different normalization.

Until the share of discrepancies reaches zero over several full trading days, there is no point in switching consumers.

The phased replacement goes from the edges to the center: first, secondary readers — reports, analytics — are moved to the new loop, then the primary ones.

The old export remains available read-only as a safety net and is turned off after the new loop has handled contentious situations: pair delisting, symbol split, exchange engine restart.

Trade collection does not stop for even a day, and each discrepancy is closed with confirmation, not by eye.

Technical doubts: latency, duplicates, request limits

Three things break trade collection from an exchange: connection loss, duplicates on reconnection, and limits on the number of requests to the venue.

All three are closed at the data ingestion level, not after the fact during report analysis — otherwise holes remain in the mart that are discovered a week later.

Ingestion is built with two channels: the streaming one delivers trades in real time, and periodic polling catches up what was missed during the window. Each record receives a composite key — venue, instrument, trade ID — and enters storage strictly once.

Repeated delivery of trades after reconnection does not create duplicates, and the discrepancy between channels is closed by reconciliation.

How do you stay within exchange request limits?

For each direction, we calculate the request weight and limit the rate with quotas; the live stream always has priority over history backfill.

When a rate limit response is received, collection pauses and continues from the same position rather than restarting — data is neither lost nor duplicated.

How do you know the data is accurate?

A gap detector runs: it marks intervals where the stream was silent longer than usual and triggers reconciliation with an independent source. The output is measurable indicators: share of missed trades, delay between a trade and its write, and share of filtered-out duplicates.

Let’s discuss the task: sources, volumes, ingestion mode

To return with a project estimate and a phased work plan, we need inputs — not in the format “we need trades from exchanges,” but sufficient to calculate the load on ingestion, storage, and processing.

Below is a checklist: the more precisely you answer the points, the fewer assumptions we will build into the calculation and the fewer discrepancies will surface at acceptance.

• which venues are a priority: centralized exchanges, decentralized pools, or both classes at once;
• which instruments and pairs are needed: spot, futures, stablecoin;
• history depth: one-time archive load for a period or continuous ingestion from the current moment;
• data ingestion mode: real-time stream, periodic exports, or a mixed scenario;
• expected volume: how many pairs and events per day, which hours create peak load;
• trade fields: time, price, volume, aggressor direction, identifiers, fee;
• how you plan to retrieve data: database, file exports, API, prebuilt reports;
• quality requirements: deduplication, gap handling, reconciliation with exchange history, storage of raw responses.

If some inputs are still open — that is fine: we will formulate assumptions and put them in a separate block so they are discussed during design, rather than surfacing at launch.

After analyzing the source composition, we send a collection diagram, storage requirements, and a phased work plan indicating exactly what is done at each step. Describe the task — we will return with an estimate and plan.