REST API development for historical data: where history gets lost
A historical market data service breaks not when it crashes, but when it responds with a success code and an incomplete body. A client requests a range of candles, receives a valid response, and builds a calculation on it — yet within the range several intervals are missing.
Such gaps in quotes are invisible in logs: no failure, no delay, just a history shorter than declared.
The cause is usually in how the response is assembled. Data is pulled page by page, hits the source's request limits, some pages arrive after retries, some with a different timestamp format.
If pages are stitched together without continuity checks, holes appear at the seams, and the time zone and timestamp rounding finish off the picture: the candle formally exists, but it is not in its correct place.
The cost to the product is not cosmetic. A backtest on a patchy history shows returns that never existed, indicators are calculated over an incomplete window, and the report diverges from the chart. Then trust in analytics collapses, and restoring it is more expensive than fixing the delivery.
We examine how this is addressed in REST API development for historical market data: a contract for the range and timestamp precision, validation of history integrity on ingest, idempotent reloading of missing data, a cache for repeated requests, and a delivery mode that does not depend on the external source's request limits.
What history export provides: completeness, speed, resilience
Completeness of history is the first thing checked at acceptance. Previously, quotes were taken from a single source, and gaps remained in the archive for illiquid instruments.
Now quote export comes from multiple sources, series are brought to a unified schema, and gaps are closed by re-requesting — a quality report shows which segments are independently confirmed and which were restored.
Response speed relies on precomputed data marts: a typical range request is served in fractions of a second, heavy queries go into asynchronous file preparation with a readiness notification. The 95th percentile of response time no longer depends on how many years of history are requested.
Service resilience is provided by caching, rate limiting of source requests, and retries with increasing delay. If an external source is unavailable, the last consistent version of the series is served, rather than an error — the terminal and analytics continue to work. An independent check is reconciliation with public aggregators, such as CoinMarketCap.
Manual edits of gaps have left the analysts' process: discrepancies between sources go into a report, not into the chart.
Customers note exactly this in reviews — “the archive is no longer a separate task: data for the required period is ready by the start of the trading day, not the middle of it.”
REST API development: history delivery options and limitations
The same quote history can be delivered in four ways, and the choice is dictated by the consumer profile: a strategy testing service needs the entire dataset at once, a trading robot needs a continuous stream, and an analyst needs arbitrary slices by instruments and periods.
We split these scenarios into separate domains: streaming subscription does not compete for connections with batch export, and heavy slices go to a storage replica and do not slow down writing new data. The difference between modes is not convenience, but which limitations you accept along with them.
Four modes and their limits
| Mode | When it fits | Limitations |
|---|---|---|
| Paginated query | Integrations, reports, backfilling missing intervals | Number of requests grows with history depth; on restart a cursor is required, otherwise you get duplicates or gaps |
| Batch export | Initial load, strategy testing, archive recovery | Data lags behind the current moment; you will have to map the file to your own schema on the consumer side |
| Streaming subscription | Online quotes, risk monitoring, on-the-fly indicator calculation | Delivers only “from this moment on”; after a connection break, catch-up via a historical query is required |
| Direct query from storage | Analytics, arbitrary slices, data reconciliation | Read-only and a separate copy; the schema is fixed, any changes go through approval |
Limitations are a property of the mode, not a defect: the stream does not provide depth, the file does not catch up with real time, the replica does not write.
Therefore, in the API description we fix the boundaries of each mode and the rules for transitioning between them — catch-up after a break, cursor on restart, data reconciliation between copies.
From source audit to handover: stages of engineering work
The project begins not with code, but with analyzing where the data comes from.
Exchange exports, aggregators, and internal archives differ in granularity, time zone, and revision rules — already at this point it is clear which series will have to be spliced and which can be served as is.
- Data source audit. We review export formats, history depth, request rate limits, and venue behavior during failures. The output is a list of sources with notes on where gaps and duplicates are expected.
- Data schema design. We define entities (trades, candles, order book), time zones, rounding precision, and series splicing rules. The schema becomes a contract: both the storage and the structure of API responses are built on it.
- Implementation. We build an ingestion layer with retries and idempotent writes so that repeated imports do not create duplicates, and a delivery layer — with cursor pagination and predictable error codes.
- Validation against a reference set. We manually label a small segment of history and compare it with the source down to the last record. We catch discrepancies before they reach analytics.
- Acceptance testing. We run the customer's scenarios: illiquid instruments, period boundaries, peak loads. We confirm that delivery fits within the agreed timelines and does not lose records.
- Handover and monitoring. We provide the schema description, configuration, and deployment procedure, and after launch we monitor data freshness and response latency.
The order is not accidental: each step relies on the artifact of the previous one — the source list, the schema, the reference. Therefore acceptance is closed before handover, not after.
What you keep: specification, schemas, procedures
A historical market data service lives longer than the team that launched it: in six months it will be developed by another shift of engineers — or yours, without the authors at hand.
That is why we hand over the working interface together with a package of artifacts that allow the service to be maintained without reconstructing the logic from code.
- Interface specification — methods, parameters, response formats, error codes, and request limits in machine-readable form: clients and tests are generated from it.
- Data schema — tables, keys, types, indexes, and history retention rules; it shows why a query by date range remains fast as volume grows.
- Update procedure — load schedule, window for backfilling missing periods, integrity checks, and reconciliation with the source.
- Access policy — roles and scopes, key lifetimes, per-consumer quotas, request log, and access revocation procedure.
- Team documentation — local launch, environment variables, migrations, metrics, release checklist, and rollback procedure.
- Automated tests and reference request examples — capture expected behavior so that changes in historical data parsing are caught before release.
- Failure instruction — what to do about gaps or slow source responses, how to switch to a backup stream, and how to confirm recovery.
Practice: how we closed gaps in history and sped up delivery
The project was an exchange data aggregator: a REST API service delivered historical candles and trades for analytics and trading robots.
At the input, up to 3% of minute candles were missing for certain periods, and assembling a one-year dataset for an instrument took about forty minutes — clients pulled history page by page and often timed out.
History recovery
Replicas were pulled from several sources, each trade was mapped to the primary key “instrument + timestamp,” and discrepancies were resolved by a source priority rule.
Missing intervals were backfilled from archival dumps and logs, and aggregates for disputed windows were recalculated from scratch. After the run, minute series completeness reached 100%, and discrepancies between sources fell to fractions of a percent.
Peak load and delivery
Under peak load, the service previously grew in latency: the request queue accumulated, and some clients did not wait for a response on long ranges. We moved hot ranges to the columnar storage ClickHouse, precomputed weekly and monthly aggregates, and limited page size.
At a peak twenty times higher than usual, delivery remains steady, and assembling a one-year dataset has been reduced to a minute.
Project result: history without gaps, delivery under peak load without interruptions, documented rules for reconciling sources, and a repeatable recovery run on schedule.
How history export differs from a real-time stream
Archive export in a REST interface responds to a range query: the client passes a time interval and instrument identifier, and the service returns a page of records with a cursor to the next one.
The result is deterministic — repeating with the same cursor returns the same rows, so a connection break is healed by re-requesting rather than restoring a subscription.
A real-time stream is arranged differently: the subscription lives in a persistent connection, and each record is delivered once.
A failure here does not repeat itself — the lost message is backfilled from the journal by the timestamp of the last received record; otherwise a hole remains in the history.
Delivery latency in a stream is measured by the tail of the distribution, not the average: the consumer notices rare spikes at peak volume immediately.
This leads to different storage requirements. The archive needs a complete array, partitioned by date, with an index by key and time — it grows monotonically, so it is moved to a cold tier where reads are more expensive than writes.
The stream needs only a short hot buffer, but the buffer must survive a node restart: without an on-disk journal, a subscription after a crash loses a window that can no longer be recovered from the source.
Reloading changes is a separate domain. In archive export, the window is deliberately overlapped by several intervals and the edges of the range are reconciled so that records at the boundary are not lost.
In the stream, reloading proceeds by cursor with an idempotent record key: duplicates are discarded, gaps are closed.
In both cases, checksums — the number of records and the boundary timestamps — are reconciled regularly; otherwise the discrepancy will surface in the customer's reporting.
Practical conclusion: a single interface can deliver both modes, but behind it are two different services with their own storage schemas, limits on query depth, and latency budget. Design begins with the question of what to do on a break — replay the range or backfill by cursor. The answer determines both the storage schema and what data reconciliation will look like.
Technical doubts before starting work
Engineering teams check failures and data edges before starting — and the questions are roughly the same for everyone: gaps, behavior when a source fails, history boundaries, reconciliation, and export. Below are direct answers, without promises that “everything will be stable.”
What should be done about gaps in a source's historical data?
We do not fill a gap silently: it is marked with an explicit completeness flag in the response, rather than disappearing between adjacent points. A range with a gap can be re-requested — repeated backfilling is idempotent, and duplicates do not appear.
A separate endpoint returns a coverage map by instruments and periods so that the consumer sees the boundaries of reliability in advance.
What happens if the source stops responding?
We serve the last confirmed snapshot together with its freshness time and a staleness flag. The consumer decides whether to work with this data or wait — instead of an empty response and error handling on their side.
Can the history depth be extended later?
Yes, backfill proceeds in ranges and chunks, and the response contract does not change — the extension does not break already written clients. New and previously loaded segments are reconciled at the seams so that no duplicate or diverging values appear at the boundary.
How is it verified that the data matches the source?
Reconciliation is performed using checksums of snapshots for sampled periods; discrepancies go into a report rather than dissolving in logs. This provides reproducibility: the same window requested again returns the same result.
Can we take our export with us?
The export is delivered as a stream to a file using the same field schema as in REST responses — the data is portable and not locked inside the service. We choose the format to fit your processing pipeline.
Next comes discussion: describe the task, and we will return with an estimate and a plan.Let's discuss the task and return with an estimate and a plan
The first conversation is needed not for introductions, but to fix the boundaries of the work: without an understanding of history depth, instrument set, and acceptable update latency, an estimate turns into a range, and the data schema turns into rework in the middle of the project. That is why we clarify what the scope of work directly depends on.
- History depth and sources: exchange feeds, vendor archives, proprietary exports — for what period data is needed and where gaps are possible.
- Data composition: ticks or aggregated candles, intervals, the list of instruments and venues, whether quote adjustment for splits and dividends is needed.
- Quality requirements: series completeness, precision to the digit, a unified time zone, rules for resolving discrepancies between sources, and duplicate filtering.
- Response format and navigation over volume: JSON, CSV, or Parquet; paginated delivery, cursors, filters by date range and instrument.
- Load and access: how many simultaneous requests the consumer supports, whether keys, permission separation, and rate limiting are needed.
- Updating and resilience: how often we load new data, how quickly we close a gap in the archive, and what the client sees when a source fails.
- Acceptance: on which datasets we reconcile the output with the reference and what share of requests we consider as fitting within the target response time.
Describe the task in free form and note what from the list has already been decided — we will return with a work estimate, data schema, and phased implementation plan.