OHLCV data from exchanges: where historical series break
Historical candles are almost never continuous. An exchange goes into maintenance, halts trading in an instrument, changes calculation rules — and a gap remains in the series.
It is invisible on the chart: the next candle simply adjoins the previous one, and no one asks whether there was trading during those minutes.
The second limitation is desynchronization between venues. Different daily close times, different trading calendars, and different instrument start times produce series that do not align at their boundaries. If you stitch them together as is, the strategy will “see” a move that did not exist in the market.
The cost of such an error shows up during strategy validation. Gaps in quotes turn into false signals, and a one-candle shift turns into a trade at a price that never existed.
Historical results stop reproducing in live trading, and the cause is sought for weeks because no one measured historical data quality.
Manual collection of quotes from exchanges does not scale. Venues change request limits, add pairs and timeframes, require keys; dozens of instruments across multiple intervals have to be verified by eye. Any correction is lost on the next export — the series is no longer reproducible.
We break down how this is closed: how OHLCV data is collected, how series completeness is checked, and what must remain in the log so the verification result can be repeated a month later.
A verified candle series: faster validation and fewer false signals
A backtest breaks not on strategy logic but on input data. A candle with lost volume, a duplicate bar when two venues are merged, or a missing interval during exchange maintenance — and validation of trading strategies shows an entry where no trade actually occurred. The cost of such an error is a rewritten trading module and distrust in all validation results.
We export candles directly through venue trading interfaces, including Binance: page by page by time, with retries on network failures. Data preparation is reduced to a single schema — UTC timestamp, open/high/low/close, volume, ticker, timeframe.
We compare bars from different sources by timestamp, discard duplicates, and mark intervals with zero liquidity with a flag so they do not enter indicator calculations as real trades.
A separate stage is series continuity. Using the trading calendar, we find missing intervals and restore them: first from a neighboring venue for the same instrument, then by aggregation from a lower timeframe. Where restoration is not honest, the bar is marked as missing, and the test window does not use it.
At the output, preparing history takes hours instead of weeks of manual collection, the share of restored gaps reaches 95–98%, and false signals at breaks stop being counted as entries — over long periods this noticeably changes the drawdown profile.
The series is reproducible: the same request to the storage returns the same result even a month later.
Methods of OHLCV data collection: full export and incremental loading
The data source determines how much time the first fill will take and how often you will have to return to connection breaks afterward.
Exchange archives provide history in packages for past periods — this is the fastest way to get a multi-year series, but files are published with a delay and require format parsing.
The exchange’s public interface provides candles on request and is limited by request rate and available history depth.
Incremental loading solves a different task: it fetches only candles that closed after the last saved one, and therefore costs several requests per instrument.
In practice, the scheme is assembled from two parts — the archive covers the past, and the collection schedule keeps the current edge of the series up to date.
| Option | When it fits | Limitations |
|---|---|---|
| Full export from exchange archives | First fill, a deep series is needed across dozens of instruments | Publication delay, different file formats, volume grows quickly |
| Page-by-page collection through the public interface | Targeted instruments and short history | Request limits, breaks on long segments, slow first fill |
| Scheduled incremental loading | Ongoing replenishment after initial loading | A failure on the last candle “sticks”; the edge needs checking and rewriting |
| Hybrid: archive plus incremental loading | A working scheme for backtesting and monitoring | Requires deduplication by candle open time and completeness control |
Control points are the same for any option: deduplication by candle open time, gap checking in the series, and columnar storage where incremental loading appends partitions rather than rewriting the entire set.
This scheme reduces the first fill from a day to tens of minutes and eliminates repeated full passes through history.
History collection procedure: from source audit to handover
History collection is not a one-time export but a reproducible pipeline: it can be restarted a month later and produce the same result.
Therefore the order of work is fixed — first we determine which sources are available and how much they can be trusted, then we describe the schema, and only then do we write the collector.
An error in the first two steps costs the most: a discrepancy in candles surfaces during the backtest, when strategy results do not match market behavior.
- Data source audit. We look at which exchanges and pairs are needed, what the real history depth is, how the source reports gaps and trading pauses, what limits exist on request rate, and where historical exports break off.
- Collection schema. We fix a single OHLCV format, the set of timeframes, time alignment to one time zone, rules for aggregating lower intervals into higher ones, and volume rounding.
- Collection setup. We configure page-by-page exports by time windows, retries on breaks, backfilling of missing sections, and idempotent writing so that a repeated run does not create duplicates.
- Candle reconciliation. We compare overlapping sections against a reference source, catch discrepancies in open and close prices, anomalous volumes, and differences in interval boundaries.
- Gap control. We check series continuity, separately handle maintenance pauses, delistings, and ticker changes, and mark sections where data was restored from an alternative source.
- Validation and packaging. We run checks for time monotonicity, zero and duplicate candles, and lay out the array by pairs and months for fast querying.
- Data handover. We deliver the array in an agreed format with a description of fields, a source log for each section, and a re-collection script — the team can update history independently.
What remains with you: schemas, regulations, ready exports
After project delivery, your team retains not only the data itself but everything needed for independent reproduction: described sources, processing rules, and working loading scripts.
We do not leave the solution in a “works, but it is unclear how” state — a support engineer will be able to bring the environment back up and get the same series.
- Source schema. A list of venues and instruments with intervals, priority when prices diverge, and the rule for selecting the reference series.
- Data documentation. OHLCV field dictionary, time zones, volume units, rules for handling gaps, duplicates, and trading breaks.
- Rebuild procedure. What to run and in what order when adding an instrument, changing the export format, or finding a hole in history.
- Exports to storage. Series in partitioned tables and flat files, suitable for loading into an analytical environment without additional cleaning.
- Quality control reports. Cross-source reconciliations, a list of anomalous candles, and coverage by date — showing where data is absent and why.
- Solution handover. Access, deployment on your side, incident review, and knowledge transfer to the support team.
- Processing version log. What changed in normalization rules and how it affects comparability of results from past tests.
Case: restoring gaps in history across 40 pairs
The client is an algorithmic trading team: it needed history for 40 pairs with minute candles over several years; otherwise, testing strategies on history produced false results.
We collected OHLCV from three sources, including Binance, and immediately saw a problem: about 3.4% of candles were missing. The gaps fell during venue maintenance and periods when there were no trades in the pair at all.
Each gap was classified rather than closed with a guess: real absence of trades, export failure, or timestamp discrepancy between venues.
For real holes, we rebuilt candles from trade archives; for apparent ones, we wrote zero volume with an explicit mark so the model would not mistake silence for a move.
Continuity control worked automatically: a gap longer than one interval went into quarantine until confirmation.
Numbers before and after. The share of gaps fell from 3.4% to 0.02% across all trading pairs. Preparing the series took about six hours of manual reconciliation — now it is a run of several minutes, and a person reviews only disputed sections. Results were rechecked in three rounds per pair — now there is one discrepancy summary.
Each series is versioned: the pair has an inventory with checksums and a cut-off date, so repeated collection reproduces the same file.
This removed a class of failures where a test diverged from live trading because of an unnoticed gap, and made it possible to compare strategies on the same dataset.
How does incremental loading differ from a full reload?
Incremental loading of OHLCV data requests only those candles that are not yet in storage — from the timestamp of the last written candle. A nightly top-up across dozens of instruments takes minutes and does not hit exchange request rate limits.
The trade-off for such savings is accumulation of discrepancies.
If a candle was not loaded because of a connection break or the exchange later clarified volume, a gap or outdated value remains in the middle of the series — and it will surface only during data reconciliation, after the strategy run.
A full reload re-requests the entire range and rewrites partitions entirely. It removes the consistency issue: volume clarifications, restored gaps, and changes in instrument composition are pulled in automatically, and the result does not depend on how previous runs performed. The cost is that time and the number of requests grow linearly with history depth.
For an already deployed storage, a mixed mode is more often used: daily scheduled incremental loading plus periodic full reload of a rolling window — the last few weeks.
Connection breaks are not scary: progress is recorded by the timestamp of the last written candle, and insertion is idempotent by the key “instrument + interval + open time,” so a repeated run continues from the same place without duplicates.
Bottom line: incremental loading is for prompt history updates, a full reload is consistency insurance on a limited window, and mixed mode is for when storage lives long and is updated every day.
Technical doubts: history depth, gaps, formats
Before collection starts, the same doubts almost always surface: whether history depth is sufficient, what to do with holes in the series, and how to later reconcile our candles with what the strategy sees.
We collect OHLCV from several sources — from Binance to regional venues — and answer each point below before work begins, not after the first export.
How deep is history and where does it break off?
It depends on the venue and timeframe. Hourly and daily candles usually go back years, while minute candles at most exchanges are provided only for the last two to three years — public interfaces simply do not expose deeper data.
Where history breaks off, we obtain an archive from alternative sources and explicitly mark the boundary so tests on that section are not considered reliable.
What to do about gaps and exchange holidays?
A gap in the series is not always an error: on weekends and on days when trading is halted, the candle does not physically exist.
We distinguish three cases — absence of trading, collection failure, and a real gap in the source — and close them differently: with zero volume, a repeated request, or a separate flag in the data.
Otherwise, the strategy mistakes a trading halt for zero volatility and shows a historical result that did not exist.
In what time zone and format do you store candles?
We bring everything to a single time zone while preserving the source’s original timestamp; otherwise, candles from different venues do not align by session open and close. Storage is columnar, partitioned by date and instrument, plus export in tabular form for manual checking.
Reconciliation is done by repeated request and volume comparison: a discrepancy between sources above the threshold is marked as suspicious and reviewed before data handover.
Let’s discuss the task: we’ll collect history for your strategy
To make the estimate based on facts rather than guesses, a few inputs are enough to start.
Send a list of venues and instruments, timeframe and history depth, and the format in which the data should land in your storage — this is enough to calculate the request volume and rate limits for each exchange.
- List of exchanges and instruments: spot pairs, futures, individual tickers.
- Timeframe and history depth — minute candles over several years or hours over months: the number of requests directly depends on this.
- Export format: set of columns, time zone, rounding, CSV or Parquet.
- How the strategy works with OHLCV data: indicators, rolling windows, rules for handling gaps and liquidity breaks.
- Storage location and update frequency: files, object storage, or database.
- Acceptance criteria: how many control points to reconcile with the primary source and what discrepancy to consider significant.
We build the work plan from a verifiable result: first a trial sample for one instrument over a short interval, reconciliation with the reference, then full loading across all venues and setup of regular updates.
This way discrepancies in time zones and missed candles surface at the beginning, not after exporting the entire volume. The timeline estimate comes together with the plan and stage breakdown so the launch order is clear before work begins.
Describe the task in free form — we will return with an estimate and plan for your strategy.