Why an order book pipeline loses data at peaks
An order book pipeline breaks not in a calm market but in the first seconds of a spike: the quote stream grows by an order of magnitude, and the bottleneck shows up where it is invisible under steady load.
Lost updates at this moment do not look like an error — the process is alive, metrics are green, the order book simply stops matching the exchange.
A session break at a peak is the most expensive: while reconnection is in progress, deltas accumulate in a buffer or are lost entirely.
The second class of failures is missed snapshots and recovery desynchronization.
If a snapshot does not match the delta stream by sequence number, the order book is assembled from incompatible layers, and after that no validation can distinguish such an assembly from a correct one.
Engineering-wise, this is treated with sequence reconciliation, re-requesting the snapshot, and rolling back the window — but only if the pipeline is designed with the understanding that updates arrive out of order and with different latency from different venues.
The third, most insidious failure is feature drift between training and inference. On historical data, the order book was assembled using one normalization method; in real time, another — and the model sees a distribution it has never encountered.
Data latency adds a shift on top: a feature calculated from a stale snapshot is formally correct but describes a market that no longer exists. Externally, this looks like degradation in prediction quality with no errors in the logs.
We examine how this is addressed: integrity checks at every transition, a unified feature calculation layer for training and inference, and data latency control along the entire path from venue to model.
What a well-built order book pipeline provides
An order book pipeline is not about moving files around; it is a data contract: every dataset has a version, a schema, and a set of checks under which it is considered fit for use. Below is what exactly changes when such a contract is fixed.
- Reproducibility. Any dataset is assembled from a version of source records and a version of feature preparation rules. A repeat run produces the same metric values, and the discrepancy between training and validation stops being a mystery.
- Input data quality. Checks catch missed snapshots, duplicates, time series gaps, and broken level ordering. The share of rejected sessions is visible in the report rather than being discovered from anomalous model metrics.
- Historical data. Book snapshots are stored in the columnar Parquet format partitioned by dates and instruments: a sample for the required period is read in minutes rather than hours of full scanning.
- Feature preparation. Order book imbalance, spread, cancellation intensity, and post-impulse recovery speed are calculated in a single pass. Manual preparation is reduced from days to minutes and stops depending on a specific analyst.
- Unified format. Differences in units, timestamps, and precision across venues are normalized at ingestion — the failure class of "data arrived but in the wrong form" goes away.
- Resilience to spikes. Buffering and backpressure keep the stream stable during activity peaks: messages are not lost, and the queue is drained after the load subsides.
Methods for collecting and storing quotes: comparison
The quote collection mode determines what the model receives: order book depth at the moment of an event, timestamp regularity, and training reproducibility.
The options below do not compete with each other — they are combined by horizon: history is covered in batches, while the live stream runs continuously.
| Option | When it fits | Limitations |
|---|---|---|
| Batch history loading | Large retrospective corpus for training and backtests | No live data, gaps between files are stitched manually |
| Continuous stream processing | Model runs online, reaction latency is critical | A connection break loses part of the deltas irrecoverably |
| Hybrid: snapshots + delta stream | History, live, and correct restart are needed | Harder to maintain: event ordering and deduplication |
| Buffering in temporary storage | Load peaks and an unstable channel to cold storage | Delay before writing, buffer eviction control is needed |
For an order book pipeline, the key criterion is time consistency. Batch loading provides volume but loses microstructure; streaming preserves it but requires a failure recovery mechanism.
A hybrid scheme with a buffer resolves this contradiction: deltas accumulate in fast storage, snapshots are recorded on a timer, and a restart leaves no gaps.
On the storage side, these are columnar files partitioned by instrument and date, plus a delta log — then the training sample is assembled in a single pass.
How we build a quote processing pipeline
The sequence of implementation stages is the same for any source — an exchange API, an aggregator, or a broker gateway. We start with an audit of sources: update frequency, snapshot format, behavior on connection loss, event numbering.
The pipeline design depends on these details — whether an intermediate log is needed or stream processing is sufficient.
- Source audit. We inventory the streams: real-time channels, order book snapshots, depth, connection limits. We separately examine fields that do not always arrive and how the source signals a gap.
- Pipeline design. Layers for ingestion, normalization, deduplication, order book assembly from snapshot and deltas, and writing to columnar storage. We define buffering points so events are not lost during load peaks.
- Collection and normalization. Unified time resolution, a shared instrument reference, agreed units of volume and trade side; exchange identifiers are mapped into a single correspondence table.
- Data validation. A run on historical segments with known anomalies: sequence continuity, absence of an inverted order book, share of recovered gaps. We investigate discrepancies against the reference down to the root cause.
- In-loop observability. Gap counters, delivery latency, share of out-of-order events — with alerts. The pipeline reports degradation itself instead of passing corrupted data to the model.
- Solution handover. Stream diagrams, configurations, restart and failure recovery procedures, team training — so that launches and changes happen without us.
The output is a reproducible order book pipeline: from raw messages to model features. Any stage can be restarted and compared against the reference, and data quality is visible from metrics, not from gut feeling.
Artifacts: diagrams, procedures, and configurations
The handover package is not a folder of files "for memory" but working documentation by which the client's team can independently deploy and run the order book processing pipeline. Below is a list of what remains with you after handover and what you use without us.
- Data schema: description of order book event fields — levels, sides, volume, timestamps, instrument identifiers, normalization and gap handling rules.
- Collection configurations: source connection parameters, snapshot frequency, order book depth, reconnection and buffering rules for connection breaks.
- Data flow diagram: the path from raw message ingestion to the feature mart, with quality control points and places where the stream can be replayed.
- Access policies: read and write roles, segregation by instruments and environments, and the procedure for issuing and revoking keys.
- Maintenance procedures: what we check daily, which latency and loss thresholds we consider a deviation, and whom to escalate to and how.
- Monitoring: a set of metrics — processing lag, share of missed intervals, clock desynchronization, queue growth — linked to notification rules.
- Recovery plan: savepoints, restart procedure from the last consistent snapshot, and verification of recovered data completeness.
Documentation lives in the same repository as the code, so procedures do not diverge from the actual configuration.
When the source or message format changes, updating the field description is part of the same work cycle — otherwise the data schema becomes outdated within a couple of weeks.
Case study: restoring a quote stream after a failure
Failures on the data source side and subscription breaks disrupted the quote stream: after reconnection, some updates arrived again, and some did not arrive at all.
The share of lost updates reached 5–7% per day, gaps appeared in sequence numbers in the order book, and resubscription added duplicates — history that could not be trusted was being used for training.
Before
Recovery after a break was done manually: an engineer restarted the session, requested a full order book snapshot from Databento, matched it against the accumulated stream, and cleaned up overlaps.
One such operation took 30–40 minutes, up to ten of them accumulated per day, and dataset reconciliation diverged in three out of ten runs. The model trained on a biased sample, and the degradation was invisible: validation metrics fluctuated from run to run.
What we did
Each update received a sequence number and a checksum of the previous state, so a break is visible immediately — by discontinuity in numbers, not by indirect signs.
The buffer holds a window of the last few seconds, the snapshot is loaded automatically and applied to the deltas, and matching records are discarded by the "number + timestamp" key before being written to storage.
Result
The share of lost updates is 0.02%, and gaps in sequence numbers are closed in the same window where they are detected. Recovery after a connection break takes 1.5 minutes versus 30–40 minutes of manual work, with zero manual operations per day.
Dataset reconciliation stopped diverging: discrepancies are caught at ingestion and do not reach the training sample, so validation metrics became reproducible.
The cause of a failure is now visible in the log along with the moment of the break and the stream segment — investigation takes minutes, not half a day. We build such a resilience loop at the stream design stage, not after the first incident.
How does an order book differ from a trade stream?
An order book is the state of limit orders: price, volume, side, and time of change. Placement, cancellation, partial and full execution generate separate messages, so the order book feed is an order of magnitude larger and updates noticeably more often than the trade stream.
The trade stream records only what has already been executed, whereas the book shows market depth: liquidity at distant levels, spread, and supply-demand imbalance.
This leads to different feature classes — trades reveal initiative and execution aggressiveness, while the book reveals price resistance and the buffer against slippage when entering with large volume.
For microstructure models, this difference is exactly what matters. The book provides liquidity and boundary pressure features, while the trade stream provides execution and short-term impulse features.
Aggregated quotes collapsed by price levels and intervals reduce storage volume but erase part of the microstructure — the choice of precision here is always a trade-off.
Choosing a source is a question of forecast horizon and feature type, not taste. Some tasks require both sources together: an individual trade without the book state is not explainable, and a book without trades does not show who exactly moved the price.
Therefore both streams are synchronized in time, and desynchronization and gaps are handled separately — otherwise the output features will diverge.
At ingestion, both sources are written separately, with timestamps and instrument identifiers, so features can be recalculated retroactively on a historical window.
Technical doubts before starting work
The doubts before starting are usually the same: how to integrate your own sources, what to do about historical gaps, and whether the existing storage will have to be broken. Below are brief answers to each point.
The pipeline is built around a normalized event schema: instrument, timestamp, side, price level, volume. Any source is brought to it by an adapter, so quality control, order book recovery, and model marts work the same way — regardless of whose data is coming in.
Can we connect our sources, not just yours?
Yes. We write an adapter for your channel or export and fix the contract of the normalized schema. After that, the source lives on equal terms with the others: the same integrity checks, the same freshness monitoring, the same marts. If you have a historical archive in your own format, we will process it in a separate pass.
What should be done about gaps in old data?
We recover the order book with a reverse pass: take the nearest complete snapshot and apply deltas up to the next one. Where recovery is impossible, windows are marked as incomplete and do not enter training — the model does not learn on gaps. All edits are written to a log so it is visible which part of history was rebuilt.
Will data be duplicated during reprocessing?
No. Writing is idempotent: the key is source, instrument, and event timestamp. A repeat pass updates the same rows rather than adding new ones, so marts do not inflate and metrics do not "drift." This also makes it safe to restart a segment after a failure.
Will the existing storage have to be rebuilt?
Usually not. The pipeline connects to the existing storage and writes to separate partitions without touching production tables. If needed, we deploy a parallel mart, reconcile the results, and only then switch consumers over.
What about support after handover?
We leave monitoring for breaks and latency, alerts for anomalies, and a procedure for backfilling missed windows. We hand over documentation and incident analysis so the team can maintain the pipeline itself.
Let's discuss your quote processing task
Estimating the scope of work for an order book processing pipeline depends not on the number of exchanges but on how the primary data is structured and what needs to be produced as output.
If we can immediately see an example export and a description of the target feature, we will come back with a breakdown by stages and data requirements, rather than a "from-to" range.
What is useful to send in advance:
- Sources: exchanges, instruments, history depth, and the required frequency of order book snapshots.
- Stream format: snapshots only or snapshots plus delta updates, how gaps and sequence repeats are marked.
- Data requirements: event schema, timestamp handling, synchronization, rules for recovering missed intervals.
- Target features: order book imbalance, spread, cancellation intensity, liquidity profile — and the horizon over which the model must operate.
- Operating mode: historical replay, real-time stream, or both, and what latency is acceptable.
- Infrastructure: where processing is hosted, what access and storage volumes are available, and whether reproducibility of dataset assembly is required.
We assemble the team composition for the task profile: a stream processing engineer, a storage specialist, and an analyst responsible for feature alignment. Describe the task in free form — we will come back with a scope estimate, an implementation plan, and clarifying questions.