Why order book scraping breaks under peak loads
Peak loads in order book scraping are the moment when an exchange emits many times more messages than in a calm market: an order book with a dozen levels turns into hundreds of updates per second.
Real-time order book collection stops fitting into the latency budget — the handler cannot apply changes fast enough, the queue grows, and the data arrives already stale.
The first thing that breaks is the connection. During volatility, the stream grows, buffers overflow, and the disconnect happens at the worst possible moment.
After reconnecting, the stream continues “as is,” without reconciling the snapshot and sequence number — and levels that no longer exist on the exchange remain in the local book.
Second is depth. Many sources provide only top levels, so during a sharp move the required depth simply is not there, and aggregated snapshots cannot restore it retroactively.
The third failure is desynchronization between sources: different timestamps, different order of applying changes, and arbitrage signals are calculated from two mismatched book states.
Therefore, order book scraping is primarily about state correctness, not read speed: snapshot reconciliation, buffering updates during reconnects, lag monitoring, and explicit loss recording. We explain how this is addressed.
Continuous market data collection: speed and completeness of the picture
Order book collection is a continuous stream, not a one-off request: dozens of updates per second for each pair, and one missed packet makes the subsequent picture incorrect.
Worst of all, the loss is discovered not at the moment of failure but during historical analysis, when the data already sits in storage with a gap.
We reconcile the update sequence with a periodic full order book snapshot and record a gap at the moment it occurs.
We connect sources in parallel — venues such as Binance and alternative channels — and compare them against each other: a discrepancy immediately shows which channel is losing data. The share of lost updates goes from tenths of a percent to zero.
Collection resilience relies on automatic reconnection, backfilling the missed window from a snapshot, and deduplicating repeats during a connection drop.
This eliminates a whole class of failures — silent update loss, when the system formally works but returns data that no longer matches the market.
We measure delivery latency along the entire path — from the exchange event to the write and to the subscriber.
Buffering and batch writes reduce reaction time to a market event to fractions of a second, and the fill rate of minute intervals in history reaches 100% instead of the previous 70–80%.
Continuous collection here is not an end in itself: strategies, risk control, and reporting rely on data completeness.
Comparison of quote collection approaches: what to choose for the task
Quote collection from the order book fits into three modes: streaming subscription, scheduled polling, and a combination of both. The choice is determined not by the “advancedness” of the channel but by requirements for data completeness, acceptable latency, and maintenance cost.
| Option | When it fits | Limitations |
|---|---|---|
| Streaming collection | All order book changes and minimal latency are required | Maintains a persistent connection, requires reconnection and backfilling changes after a drop |
| Periodic polling | A snapshot once every few seconds is sufficient, and there are many sources | Misses bursts between polls, hits request limits |
| Hybrid scheme | Stream completeness and resilience to drops are required | More complex coordination of two channels and elimination of discrepancies between them |
Streaming collection gives a complete picture of events, but any connection drop leaves a “hole” in history — it is closed by re-downloading a snapshot and pulling changes for the missed interval.
Scheduled polling is simpler and more reliable to maintain, but it loses microstructure: liquidity bursts between ticks simply do not make it into the accumulated data.
The hybrid scheme takes a snapshot via polling and then maintains the stream, and on a drop it recovers from a reference point.
We run the modes on the customer’s real stream — calculate the share of missed changes and recovery time after a drop, and only then finalize the scheme.
Stages of implementing a data collection and normalization system
Order book scraping does not break at the start but after weeks of operation — when a venue changes format, the connection drops, or update numbering diverges.
Therefore, implementation proceeds through a sequence of stages, and each is closed with a verifiable artifact: without it, the next step makes no sense.
- Source audit. For each venue, we record the field composition, order book depth, update frequency, connection limits, and reconnection rules. The output is a table of sources with priorities and caveats.
- Stream schemas. We mark out layers: raw messages, normalized records, historical storage. We keep raw data separately so processing can be replayed without re-collection.
- Collection setup. Connectors with automatic recovery after a drop: snapshot plus replay of deltas from the last confirmed number, lag monitoring, and clock synchronization.
- Normalization. A unified schema — instrument, side, level, volume, price precision, time in UTC. Discrepancies in tickers and number formats are closed with a mapping dictionary.
- Data completeness check. We reconcile the stream against a control snapshot, looking for missed sequences and gaps. The share of recovered gaps and reaction time go into monitoring.
- Handover to operations. Restart procedures, dashboards, alerts for gaps and lag growth, access, troubleshooting of typical failures — with a warranty support period.
What we hand over to operations: configurations, schemas, procedures
After launch, the collector remains with you, not under our support. We hand over everything needed for independent operation of the stream: source configurations, data flow schemas, access rules, documentation, and support procedures.
We close the artifacts at acceptance: your engineer deploys collection in your environment, connects a new source, and restores data after a drop without our help. Below is what exactly is included in the package.
- Source configurations — exchanges and channels, request rate limits, authorization schemes, priorities during drops; a new source is added by editing a single file.
- Stream schemas — subscription topology, buffering at peak loads, the order of reconnection and assembly of a snapshot with subsequent updates.
- Access rules — roles, key separation, rotation, access log; historical archives are isolated from the production environment.
- Data dictionary — unified field names, mapping of tickers to instruments, rules for deduplication and handling incomplete snapshots.
- Documentation — message format, failure triage procedure, typical diagnostic scenarios, and a list of stream health metrics.
- Support procedures — target recovery times after a drop, the procedure for connecting and disconnecting sources, and archive retention policy.
Case study: how order book collection stopped losing data at peaks
The customer is a market maker team that quotes on several venues, including Bybit. Order book collection was handled by four independent scripts: each wrote to its own file, started manually, and knew nothing about its neighbors.
In a calm market, the data matched, but during sharp moves the streams diverged, and some events disappeared without a trace.
What it was before
A missed fragment broke not only the chart: it distorted the calculation of spread and imbalances on which trading decisions were based, and this was noticed only after the fact.
There was nothing to verify losses against — no complete book snapshot for the moment of failure was stored. Investigating each case took hours of manual work and still did not give confidence that exactly one segment was lost rather than three in a row.
What we changed and what we got
We assembled a single pipeline: book snapshot plus change stream, an on-disk buffer in case of a connection drop, automatic reconnection with catch-up of the missed segment, and checksum reconciliation after each recovery.
Deduplication is by event identifier, so repeated delivery does not corrupt history, and a discrepancy with the snapshot becomes a signal rather than a surprise during analysis.
| Metric | Before | After |
|---|---|---|
| Data completeness at peaks | 60–70% of events | 99.9% |
| Recovery after a drop | manually, tens of minutes | automatically, seconds |
| Integrity check | selectively, after the fact | after each recovery |
How does streaming order book collection differ from periodic polling?
Periodic polling takes an order book snapshot on a schedule: the client requests depth at a given interval, receives a snapshot, and that is all.
Between two requests, the life of the order book is invisible — an order placed and canceled within the window does not remain in the data. The less frequent the polling, the cheaper the collection and the more the sample diverges from the real event stream.
Streaming collection maintains a persistent channel and receives every change: an initial snapshot arrives, then only deltas with sequence numbers.
The numbers provide end-to-end verification: a sequence gap means data loss, the connection is rebuilt, and the snapshot is requested again. This produces a recoverable order book suitable for microstructure calculation, not just average levels.
The modes hit infrastructure limits in different ways. Polling is constrained by venue rate limits — for public sources such as Coinbase, they are tied to the key and address, so collection across hundreds of instruments requires request sharding.
Streaming is cheaper in the number of connections but more expensive in processing: it needs a layer that survives drops and does not lose deltas.
The choice of mode is determined by the task: where event order and order lifetime matter, only streaming works, while regular liquidity snapshots are covered by polling. That is why we more often build a hybrid and describe how it is deployed for your instrument set.
Technical concerns: gaps, duplicates, source limits
An order book stream is a snapshot of the book plus deltas with sequence numbers. Once the connection drops, some changes are lost: the local book diverges from the source, and trading signals begin to rely on nonexistent liquidity.
That is why the collector is built around sequence reconciliation and checkpoints, not around a simple subscription.
Below are answers to engineering objections that come up during architecture evaluation.
The connection dropped — how do you restore state?
Each stream writes a snapshot and sequence position to persistent storage. After reconnecting, the service loads the last checkpoint, reads deltas from it, and reconciles the number with the source.
If the gap is wider than the change retention window, the book is rebuilt from a fresh snapshot, and the omission is recorded in the log with an explicit marker.
How do you exclude duplicate records?
The idempotency key is source, instrument, and change number. The write is an upsert by unique index, so repeated delivery after reconnection does not create second rows. For trades, the same approach works by trade identifier.
The source limits the number of requests — what then?
Limits are planned in advance: requests go through a queue with a budget per interval, on failures an adaptive pause between retries is enabled, and instruments are divided into priority and background. We move historical downloads to a separate window so they do not compete with the real-time stream.
How do you know the data has not diverged?
We monitor lag to the source, the share of book rebuilds, the number of missed sequences, and queue fill rate. A threshold on any metric raises an alert before the discrepancy reaches trading logic.
Let’s discuss the task and come back with an estimate and work plan
To come back with a timeline estimate and a working plan rather than general considerations, we need a minimum of inputs on the task. Real-time order book collection depends on details: the set of venues, history depth, latency requirements, and recovery rules after drops. Below is what is worth preparing before work starts.
- Venues and instruments. Which exchanges and pairs are needed, and what is more important by priority — this determines the number of parallel streams and storage volume.
- Data depth and composition. Only best prices or a specified number of levels, whether full order book snapshots and subsequent changes are needed, trades, and event timestamps.
- Latency requirements. From the appearance of a change in the order book to the write to your storage — and the acceptable share of missed updates.
- Access. Public channels or private keys with limited permissions, venue limits on request rate and number of connections.
- Where to deliver data. A stream to your bus, database writes, files, a mart for analytics; whether historical recalculation and reconciliation with exchange data are needed.
- Infrastructure. Your own servers or cloud, deployment regions, headroom for the communication channel and disk for history.
- Failures and observability. What we do on a connection drop, how we catch order book desynchronization, who receives the signal, and who responds.
Describe the task in free form — we will come back with a timeline estimate, work plan, and collection scheme that marks control points, recovery procedure, and reporting composition.