Turnkey Development of an Exchange Data Normalization System

Crypto Exchange Data Normalization: Where the Flow Breaks

The same trade on two venues arrives with different fields.

In some places, the trading pair is one field; in others, two independent tickers; volume is either in the base currency or in the quote currency, and the trade side is sometimes already reduced to “buy/sell,” sometimes provided as an initiator flag.

Different quote formats diverge most strongly on timestamps: some venues return milliseconds, others seconds, some in a local time zone instead of UTC, and the publication of a trade and its arrival in the stream are separated in time.

If you aggregate as is, candles for the same interval won't match across venues, and an event will end up in the wrong window.

Then manual trade reconciliation begins: an analyst consolidates exports in a table, matches identifiers, recalculates volumes, and manually investigates discrepancies in venue exports.

A day of history across several venues takes hours, and the result still depends on whether a person noticed a gap.

The most costly thing is losing history during a failure.

If the collector does not persist a record before passing data onward, a process crash leaves a hole, and the venue does not provide trades retroactively — the gap in the series remains forever and corrupts any calculations at its boundary.

That is why crypto exchange data normalization is built at the input, not at the reporting stage — let's break down how this is closed.

What a Unified Normalization Layer Provides

A unified normalization layer brings all venues to a single schema: instrument, trade side, price and volume in the base unit, timestamp in a common time zone, execution type flag, and a breakdown of partial closes.

Conversion happens at the input, so analytics, trading modules, and reporting read the same quote format — without separate adapters for each exchange.

Kafka serves as the transport: partitioning by instrument preserves event order, and reprocessing does not destroy already recorded history.

Connecting a new venue comes down to a connector and a field map, not rebuilding the storage or modifying consumer code.

In practice, this is days of work instead of weeks: a new source is brought up in a test environment, compared against a reference over a historical segment, and connected to the common stream.

Automatic source reconciliation covers the bulk of records — from 97% to 99%, depending on the venue. Spot and derivatives of the same pair are matched by instrument and time, and discrepancies are caught by volume checksums per minute.

The remainder goes into a queue for review: this is selective anomaly checking, not a stream of manual reconciliations that used to consume the duty shift's time.

The integrity of trade history rests on idempotent writing by the key “venue + trade ID” and an immutable log of raw responses. Normalization can be replayed from scratch and produce the same result — this saves you when refining rules or reprocessing a missed segment.

At the same time, a whole class of errors disappears: time zone desynchronization, different designations of the same pair, lost partial fills, and duplicated trades when collection restarts.

Quote Collection Methods: Comparison of Approaches

An exchange does not provide quotes in a single way: freshness and historical depth come from different channels.

A streaming quote subscription maintains a persistent connection and delivers changes as they appear — quote arrival latency is measured in fractions of a second, but the channel buffer is limited, and if the connection breaks, the missed interval must be backfilled separately.

Batch history exports provide depth of trade history and a full order book snapshot for past periods, but they are generated in chunks and lag noticeably behind the current moment.

Scheduled polling occupies the middle ground: it is predictable and does not require a persistent connection, but in a volatile market, between two requests, a move can occur that we would have seen immediately in a stream.

What We Combine

Compatibility is achieved not by choosing one approach, but by combining them: the stream maintains the hot layer and operational dashboards, exports backfill history, and polling provides insurance in case of a break.

The key condition is a unified instrument and time model; otherwise, the same trade arrives from different channels with different identifiers and does not get stitched together.

Implementation Stages: From Source Audit to Handover

We structure the order of work so that every decision is confirmed by the source's behavior, not by assumption.

First, we record what the venues actually return, then we assemble the target schema, and only after that do we write code — otherwise, rework will cost more than the analysis itself.

1. Exchange source audit. We examine trade streams, order book, and history exports: response formats, update frequency, pagination, request limits, behavior on connection loss. We separately flag sources that return data with a delay or recalculate retroactively.
2. Discrepancy map. We consolidate venue differences into a single table: instrument identifiers, price and volume precision, time zones and timestamps, trade side designation. It also becomes a working document when adding a new exchange.
3. Record reconciliation rules. We describe a unified event schema: key, symbol normalization, rounding, marking of duplicates and reopenings. We version the rules — an edit does not break already collected datasets.
4. Assembly. We implement ingestion, normalization, and writing to storage with idempotent processing: a repeated run does not create duplicates, and a gap is closed by backfilling history.
5. Validation on historical data. We run the rules on the accumulated archive: compare event counts and volumes, seams on restart, recovery after gaps. We investigate every discrepancy down to its cause rather than smoothing it with a filter.
6. Handover of the system to the client. We provide configurations, the storage schema, data quality control procedures, and the sequence of actions in case of a source failure, and train the team. After handover, the system operates and evolves through the client's efforts — our support remains an option, not a condition.

What We Hand Over: Configurations and Procedures

We hand over not only a working environment but also a kit that can be used to evolve it without us.

Each artifact is tied to a specific operation: connect a new exchange, investigate a discrepancy in a candle, change the responsible person, or understand why a source dropped overnight.

• Connection configurations — for each exchange: list of trading pairs and streams, request limits, reconnection scheme, behavior on desynchronization and missed messages.
• Normalization schema — unified instrument and trade event model, rules for bringing tickers and timestamps to a single scale, handling of renames and splits.
• Mapping dictionaries — tables between exchange and internal identifiers and the procedure for adding a new source without modifying code.
• Access rules — roles for reading data, modifying mappings and configurations; change log so that every intervention is explainable.
• Support procedures — response to a new listing, format change, and source disconnection, reconciliation frequency, escalation points.
• System documentation — architecture diagram, data flow description, exchange connection checklist, and list of known limitations.
• Acceptance set — test events with expected normalization results, so that discrepancies are caught by validation, not weeks after launch.

We separately record what remains on your side: access keys, contacts with exchanges, history retention policy.

Case Study: Consolidating Quotes from Four Venues

The client is an algorithmic trading fund operating on four venues, including Bybit.

Each returned trades and order book data in its own form: time in seconds and milliseconds, for some venues local, without offset, and the same instrument was written as BTC/USDT, BTCUSDT, and XBTUSD.

Reconciliation was manual: an analyst collected exports into tables and found discrepancies already in the finished report — by volumes, time, and missed intervals. One trading day took about six hours, and at that volume, rare but significant anomalies were missed.

The foundation was a unified data model: time is brought to UTC, instruments to a common dictionary with symbol conversion, trades are deduplicated by venue ID, and the order book snapshot is merged with subsequent changes. Each record stores its source, so a discrepancy can be traced back to the original message.

Checks for gaps and desynchronization are performed at the input, not in the report: a record with an incomplete event chain is flagged and goes into a separate review queue.

The report is generated on schedule, and manual reconciliation has turned from a mandatory step into an exception.

What If the Exchange Already Has Its Own Delivery Channel?

There is no need to break a working channel. If an exchange already has its own quote delivery channel, it becomes another receiver in the common scheme: the normalization layer is placed on top of it and brings the stream to a unified form rather than replacing the source.

Technically, it looks like this: the receiver reads what the existing environment already returns — a streaming connection, export files, or FIX — and publishes events into the internal model.

In it, each event has a fixed instrument, type, timestamp with source-level precision, and origin flag, so the calculation module, archive, and dashboard do not know and should not know which channel the data came from.

Then the coexistence of two environments and gradual migration of sources begins.

The new environment and the custom solution run in parallel: discrepancies in volume and prices are calculated automatically, and a source is switched to the normalized branch only after confirmed convergence.

The order is one source at a time, with the ability to revert the previous route without stopping collection.

Retiring the custom solution does not happen “in one day,” but as sources are covered: first, writing is removed from it and it remains in read mode for reconciliations, then it is switched off.

This way, integration into the existing scheme does not turn into a one-time risky migration — the cost of an error is paid by one source, not the entire quote collection.

Technical Doubts: Review of Common Questions

Venues change response schemas without announcements: a field is renamed, nesting is shifted, a number is turned into a string. It is not ingestion that breaks, but parsing — and then everything diverges: orders, trades, candles, candle aggregates.

Therefore, we describe the input contract declaratively: schema, field types, requiredness, conversion rules. Discrepancies go into quarantine, not the main dashboard. Below are the questions most often asked by the client's engineers.

How do you align timestamps from different venues?

We store two values: the event time by the venue's clock and the reception time on our side. We bring timestamps to a single time zone, parse whole seconds, milliseconds, and microseconds, and separately calculate and compensate for network latency. Candles assembled from different sources no longer diverge from one another.

What happens if a source fails or the connection breaks?

Reconnection with increasing delay, alongside a backup polling transport and an on-disk buffer so that a missed interval is not lost. Retries are suppressed by the record key, so duplicates do not accumulate. A gap is recorded in the completeness report rather than going unnoticed.

Do you connect a new source and reprocess history?

The source adapter is assembled according to the common contract; the instrument dictionary and ticker matching rules are described separately.

History is backfilled in chunks with checksum validation, and writing is idempotent — a repeated run for the same period does not create duplicates and matches the source data.

Let's Discuss Your Task and Come Back with a Plan

To discuss the normalization task concretely, we need a list of quote sources and the current data delivery scheme at the input.

The more specific the list of venues, instruments, and required message types, the more accurate the estimate of the scope of work — without assumptions that later turn into rework. Send what you have: a draft, a field mapping table, a configuration fragment.

Then we review point by point:

• List of quote sources — venues, number of instruments, whether trades, order books, indices, and timestamps are needed.
• Current acquisition scheme — direct connection, third-party aggregator, exports, availability of historical archives.
• Target format and consumers — where we deliver data and which fields are mandatory for the trading system, analytics, or storage.
• Latency and completeness requirements — acceptable quote lag, history depth, behavior on connection loss.
• Instrument specifics — non-standard tickers, different price steps and lot sizes, rounding rules on each venue.
• What already works — table schemas, queues, custom normalization scripts: we will preserve what does not interfere and will not break a working stream.
• Constraints and timelines — access, checks on your side, commissioning date, who accepts the result.

After the review, we will come back with an estimate of the scope of work, an implementation plan, and timelines by stage, with acceptance points at each.