Collecting Historical OHLCV Data from Crypto Exchanges

Parsing Historical OHLCV Data from Exchanges

Collecting historical OHLCV data from crypto exchanges requires a well-thought-out architecture. We've tackled this task dozens of times: different platforms, limits, formats. Manual parsing via a browser leads nowhere. You need a system that collects data from Binance, Uniswap, and 10+ other exchanges into a unified TimescaleDB without hitting rate limits. One of our projects processed 50 instruments in real time—without a single IP block, achieving 99.99% uptime over 6 months.

Why a Rate Limiter Matters for OHLCV Parsing

Naive sleep(100) between requests is a recipe for data loss and IP bans. Binance allows 1200 requests per minute per IP. If you collect 100 instruments simultaneously, you'll exceed the limit in seconds. A rate limiter with dynamic adjustment is the only working approach. We implemented a RateLimiter with separate buckets for each exchange. It consistently holds 20 req/s without overshoot, reducing API errors by 95% compared to fixed delays.

CEX vs DEX: Different Sources, Different Logic

CEX is the standard: clean candles, minimal gaps. DEX requires event indexing and price calculation via sqrtPriceX96. Our collector unifies both approaches under a single interface, processing 1000+ events per second.

How to Collect OHLCV Data from DEX Without a Native API

For DEX we use indexing of swap events via The Graph or direct RPC. Events contain sqrtPriceX96, from which price is derived. We then aggregate them into candles of a given timeframe. This allows getting data for any Uniswap V3 pool. Storage in TimescaleDB with hypertables enables fast time-based aggregation; queries run 10x faster than on plain PostgreSQL.

Stages of Collecting Historical OHLCV Data

1. Analysis – determine the list of exchanges, instruments, timeframes.
2. Design – database schema (TimescaleDB hypertable), collector architecture.
3. Implementation – write parsing scripts, rate limiter, logging.
4. Test – load 1 million candles, check for gaps, speed, duplicates.
5. Deploy – to your server or cloud (AWS/GCP), configure alerts.

How We Do It: Stack and Implementation

The main tool is the ccxt library for 100+ exchanges. On top of it, our own layer of rate limiting, logging, and incremental loading. Key components:

• Rate Limiter with token bucket (1200 req/min for Binance, adjustable per exchange)
• Multi-exchange collector on ccxt with retry and fallback support
• Storage on TimescaleDB with hypertables, compression (7x storage reduction), and continuous aggregates

The rate limiter and collector code are in the original article. Data is written to TimescaleDB, where daily aggregations are automatically created via Continuous Aggregates. Our collector is 3 times faster than the standard approach thanks to parallelization and dynamic limit management.

Comparison of rate limiting methods:

For high-load scenarios we use sliding window log, recording each request's timestamp in Redis. But for 95% of tasks, our RateLimiter suffices.

const rateLimiter = new TokenBucket({ capacity: 1200, fillRate: 1200, // per minute tokens: 1200 }); 

What's Included in the Work

• Documentation – description of the DB schema, API methods, deployment instructions
• Code – scripts for collection, update, aggregation (TypeScript + SQL)
• Access – TimescaleDB setup, user creation
• Training – 2 one-hour sessions for your team
• Support – 1 month after deployment (response within 24 hours)

Estimated Timelines

From 1 to 3 weeks depending on the number of exchanges and history size. Cost is calculated individually—contact us for an assessment of your project. This eliminates subscription costs for third-party APIs (up to $500/month) and reduces engineering research time. Get a ready-made solution for your backtesting.

Typical Errors in OHLCV Collection

• Ignoring rate limits – IP block and time loss (e.g., 12 hours downtime)
• Storing in CSV – impossible to efficiently aggregate and search; our hypertables reduce query time by 90%
• Lack of alerts – data gaps go unnoticed for weeks; we alert within 1 minute of detection

We eliminate these problems at the design stage. 10+ OHLCV collection projects—our experience guarantees a stable pipeline. The concept of OHLCV originates from traditional exchange analysis (Wikipedia). Request a consultation, and together we'll design a solution for your needs. Contact us for a project assessment.