Why WebSocket scraping breaks under peak loads
In a calm market, the trade flow is steady, and WebSocket scraping seems trivial: subscribe to channels, write to the database.
Failure comes during a spike — the exchange under load closes connections, and connection limits prevent keeping a separate channel for each trading pair.
An exchange connection drop is an operating mode, not a failure.
While the client reconnects, hundreds of events pass through the channel: trades are missed, and the order book is assembled from a stale snapshot and fresh changes — order book desynchronization, in which the calculated price diverges from the actual one. Most dangerous is that the discrepancy is not obvious: the numbers look plausible.
A separate problem is duplicate records. After a disconnect, some messages arrive again, and without deduplication by event ID, database volume grows and trade volume aggregates are inflated.
The error accumulates silently and surfaces in analytics, reporting, or the risk model, where it does not belong.
We break down how this is addressed: order book snapshot recovery with checksum verification, event buffering during a disconnect, deduplication by key, and distributing subscriptions so that connection limits cannot be hit.
What a continuous stream of data from exchanges and networks provides
A continuous quote stream instead of scheduled polling removes the main problem — price movement between requests.
When a subscription remains active, every order book change and every trade arrives at the moment it appears, and the share of lost events is measured in tenths of a percent, not in percent.
On Binance and other large venues, the stream is provided through a gateway with a limit on the number of simultaneous subscriptions — it is this limitation that shapes the collector architecture, not market volume.
A connection drop is a normal operating mode, not a failure: a limit triggers, an access key changes, a network segment goes down. The engineering difference is whether the stream recovers on its own and from what point.
We store the ID of the last confirmed event and, after reconnection, backfill the missed interval from history — connection recovery time is reduced to seconds, no operator is needed, and duplicates are filtered out by trade key.
Completeness of trade history is verified by reconciliation with an independent source: the order book snapshot and archive log serve as the reference, and the discrepancy is closed by a backfill window.
The history remains reproducible — metrics are calculated retroactively from it, including the minutes when the collector was silent.
Delivery latency must be predictable: what matters is not the average but the tail — as the number of instruments grows, it must not widen.
Collector load grows with the number of subscriptions, not with market activity: splitting instruments into independent groups, batch writing events, and a backpressure queue keep consumption steady.
Therefore the stream scales with new pairs and networks without reworking the core — another segment is added, not a second collector.
Comparison of market data collection modes
There is no single "correct" collection mode for market data: the scheme is chosen based on what is more critical — speed of reaction to an event or completeness of history over a period.
A persistent subscription gives reaction at the moment the event appears but does not survive a connection drop; periodic polling is resilient to interruptions but lags and hits exchange request limits. Below are four options that teams actually choose between at project start.
| Option | When it fits | Limitations |
|---|---|---|
| Persistent stream subscription (WebSocket) | Live order book, trade feed, reaction to an event immediately after publication | A connection drop causes historical gaps: without checkpoints and proper resubscription, some events are lost permanently |
| Periodic polling | Rare metrics, instrument reference data, scheduled state reconciliation | Delay between requests, rate limits; as the number of instruments grows, the number of requests grows too |
| Hybrid collection scheme | Trading and analytics systems that need both speed and history completeness | Requires idempotent merging: deduplication, checkpoint storage, reconciliation of discrepancies between sources |
| Own network node | Need data that public providers do not expose: mempool, full contract logs in EVM networks | State synchronization and storage, updates and on-call duty, but no third-party limits or external points of failure |
In practice, the working combination is a persistent stream subscription plus backfilling gaps using checkpoints; an own network node is connected where public sources do not provide the required data depth.
We determine the set of modes by the task and fix it in the data schema before development starts — this is more efficient than reworking storage when gaps are discovered in history.
WebSocket scraping: implementation and handover stages
Live data from exchanges and EVM networks exists only at the moment of publication: price, order book, new blocks, and contract logs do not restore retroactively by themselves. Persistent connections instead of periodic polling give minimal lag but require discipline in recovery.
Implementation is therefore built around two risks — losing an event on a disconnect and not noticing that loss. The order of work is below.
- Source analysis. We determine which channels are needed: trades, order book, new blocks, address logs. For each, we clarify limits on the number of simultaneous subscriptions and history depth — where the stream is self-sufficient and where additional data is needed.
- Data flow schema. We trace the message path: source → subscription → normalization → queue → storage → consumer. We mark buffering points so that a spike in market activity does not turn into lost events.
- Subscription setup. We set filters, publication frequency, and the resubscription rule for each channel. The consumer receives the needed events, not the entire stream — processing is offloaded without losing significant data.
- Handling disconnects. We describe behavior on disconnect: retries with increasing delay, connection liveness check, recovery from the last processed block or cursor. Missed data is backfilled from history; already received data is not duplicated.
- Data completeness check. We reconcile the stream with an independent source and history, calculate the discrepancy by number of events and the maximum processing lag. Deduplication is by event ID, not by arrival time.
- Solution handover. We provide subscription configurations, flow schema, message formats, and the failure response procedure. The client's team can see where to look at metrics and how to bring the stream up after a failure.
What the client retains after stream implementation
A stream on WebSocket subscriptions is not just a working collector, but a kit that the client's team is left alone with.
Therefore the final package includes configurations, schemas, and procedures: using them, the stream can be brought up from scratch, expanded, and handed to another engineer without verbal explanations.
- Subscription configuration — list of channels and instruments for each exchange and network, requested history depth, resubscription intervals, and reconnection parameters. Versioned; changing a subscription does not require code changes.
- Flow and storage schema — route from entry point to raw archive and analytical layer: where normalization is performed, which fields are mandatory, how discrepancies between sources are recorded.
- Access rules — roles for read and administration, key issuance and rotation procedure, separation of production and test environments, access revocation when an employee changes.
- Recovery procedure — actions on connection loss or node failure: automatic reconnection, backfilling the missed range, manual restart, and data integrity check for the period.
- Collector documentation — what exactly is collected, which metrics are captured, by what signs stream degradation is visible, and where to look when investigating an incident.
- Procedure for adding a new source — checklist: how the source differs in format and limits, what is checked before putting it into operation, how long connection takes.
Separately, we hand over the configuration change log and load report — showing when subscriptions changed and what result this led to.
Case: exchange stream without gaps and connection drops
The client is a trading team that collected trades and order book changes from several venues for signals and historical analytics.
The stream from Binance was retrieved by periodic polling: on a calm market it looked workable, but during volatility spikes some events did not make it into the sample, and the collector crashed due to a network error and waited for a manual restart.
We recorded the metrics before work, switched collection to a persistent WebSocket connection, and compared the indicators after.
| Metric | Before | After |
|---|---|---|
| Share of missed events | up to 1.8% during spikes | below 0.01% |
| Collector downtime | up to 40 minutes per day | seconds for reconnection |
| Manual restarts | 6–8 per day | 0 |
| Completeness of restored history | 92% | 100% |
Internally, instead of scheduled polling — a long-lived connection with a subscription to the required channels and writing each event to a disk queue until processing is confirmed.
Connection silence is detected by the absence of heartbeat signals, the disconnect is closed by reconnection with increasing delay and backfilling the missed range from exchange history: the last confirmed event ID is saved, so the backfill window is known exactly. Writing is idempotent — repeated backfill does not create duplicates.
Result: history without gaps is suitable both for analytics and for testing strategies on past data, and the on-call engineer no longer monitors collector liveness.
Reconciliation of daily snapshots with exchange data shows a discrepancy within rounding, whereas previously gaps were discovered only after the model had trained on incomplete data.
How does a stream subscription differ from periodic polling?
Periodic polling — the client itself requests the state at a given interval: list of orders, recent trades, balance. It is trivial to implement, does not require holding a connection, and therefore often remains a fallback channel.
The trade-off is latency: an event enters the system no earlier than the next cycle, and everything that happened between two snapshots is lost.
Event subscription via WebSocket flips the scheme: the connection is opened once, and the source itself sends a message at the moment of change — order placement, trade execution, new block release.
Reaction speed is determined by signal transmission time, not by half the polling interval. For arbitrage and reaction to liquidations, the difference is fundamental.
Load on the source is structured differently. Polling creates a steady stream of requests regardless of market activity: in quiet periods almost all return an empty response, but quotas and limits are consumed.
A subscription stream scales with activity — a spike produces many messages, and in calm conditions the channel is silent.
Completeness of the picture is the second dividing line. Polling returns a snapshot, and an order placed and canceled within the window will not appear in the data at all.
The stream preserves the sequence of events but requires disconnect handling: on reconnection, missed data is backfilled separately, and the local order book is restored — otherwise the picture will diverge from the exchange.
Therefore the models are not opposed but combined: the subscription maintains the stream, and rare polling reconciles the state and catches discrepancies. We explain how this is assembled for a specific task.
Technical concerns before starting work
Concerns at the start rarely relate to architecture — more often they are about how the new collector coexists with the already operating environment and who is responsible for incidents. Below are questions asked before signing the work plan.
We already have a collector deployed — start from scratch?
We start with an audit: we look at which data streams are already running, where messages are lost, and how connection drops are handled. Usually it is more efficient to extend normalization and history backfill alongside the current environment and switch sources one by one. The old collector works as a backup until data matches in reconciliation.
How do we avoid hitting source limits?
Source limits are a restriction on the number of active subscriptions and resubscription frequency.
The scheduler keeps a margin across streams, distributes them across connections, and when a limit is reached gives priority to critical instruments and networks, queueing the rest.
After a disconnect, the collector restores state from the nearest snapshot and catches up missed data rather than starting over.
Who is responsible for failures on the exchange side?
We collect telemetry for each source: delivery latency, disconnect frequency, delta discrepancy with snapshot. If a failure is confirmed on the source side, it is an external incident, and we record it separately from errors in our environment. Recovery is automatic: reconnect, resubscribe, history backfill.
How do we expand the list of networks and venues?
Each new source is connected via a separate adapter: its own subscription schema, mapping to a common data model, enabling behind a flag. We do not touch already running sources — expanding the list does not create risk for current collection.
What is included in knowledge transfer?
Flow and subscription schemas, configurations, on-call and incident investigation procedures, and a walkthrough with your team on real failures. After handover, you can add sources and change priorities without us.
Let's discuss the task and come back with an estimate and plan
To make the conversation concrete, just send what you already have: the list of exchanges and networks to cover, and what data is needed from each — trades, order book, transaction statuses, contract events.
Add the update frequency, required history depth, and where the result should go: to a database, a queue, or directly to your service.
In response, you receive a point-by-point breakdown that can already be used to make a decision.
- Task statement — which streams and from which venues we bring up, what we consider the source of truth, where event loss is acceptable and where it is not.
- Solution design — how connections, resubscription to instruments, and backfilling the missed interval after connection loss are arranged.
- Stage plan — what we start with, which intermediate result we consider working, at which step we validate with real data.
- Artifact composition — services, configurations, flow and storage schema, launch instructions, set of metrics and logs.
- Acceptance criteria — stream continuity under load, behavior on disconnect, historical data completeness, discrepancy with exchange data.
- Scope boundaries — what we cover, what remains on your side, and which external dependencies affect timelines.
This set is also convenient to use as a checklist when choosing a contractor: if the response has no schema, stages, and verification procedure — there is nothing to evaluate.