Your trading bot runs on servers close to the exchange. Risk management needs the same data in a data center across the ocean. Simple file transfer doesn't work: latency grows to seconds, and data loss reaches 8% of ticks over an unstable channel. We encountered a case where a client lost up to 5% of ticks due to an unstable link between New York and Tokyo. You need distributed real-time replication that can handle hundreds of thousands of messages per second without losing a single tick. Replication based on Apache Kafka reduces losses to zero and ensures consistency across all nodes. Order replication that won't fail.
We design and deploy market data replication systems turnkey. Our stack is Apache Kafka as a reliable backbone, MirrorMaker 2 for cross-regional replication, and Confluent Schema Registry for format evolution. Over the past 10+ years, we have delivered over 30 projects for crypto funds, market makers, and prop trading firms. Result: reduce infrastructure costs by up to 40% and cut downtime by 80%. Clients save an average of $15,000 per month on infrastructure by consolidating streams.
In this article, we'll break down how to build a replication system, which topologies to choose, how to guarantee consistency, and how to avoid typical pitfalls.
Why Replication Is Needed
A trading system consists of several components running in different environments:
- Production trading — co-location near the exchange, minimal latency.
- Research/backtesting — data center with large storage volumes.
- Risk management — isolated network with restricted access.
- Analytics dashboards — accessible to a wide audience.
Each environment must receive an identical data stream without overloading the source. Replication solves this by ensuring consistency, fault tolerance, and scalability.
Replication Topologies
| Topology | Description | Reliability | Latency |
|---|---|---|---|
| Hub-and-Spoke | One primary aggregator collects data from exchanges; replica nodes subscribe | Low (single point of failure) | Low |
| Chain Replication | Data is passed along a chain: exchange → primary → secondary → tertiary | High (no SPOF) | High (cumulative) |
| Pub-Sub (Kafka) | Primary writes to Kafka; consumer groups read independently | Very high (topic replication) | Medium (depends on mirror) |
For production, we recommend Pub-Sub via Kafka — it's a flexible option that scales easily. Apache Kafka provides durable, scalable, and fault-tolerant message exchange.
How to Ensure Consistency During Replication
Consistency is the key challenge in distributed replication. We solve it with a combination of idempotent consumers and atomic transactions. Consumers deduplicate messages by keys such as trade_id or update_id. For critical streams (risk management), we use exactly-once delivery via Kafka Transactions.
from confluent_kafka import Producer producer = Producer({ 'bootstrap.servers': 'kafka:9092', 'enable.idempotence': True, 'transactional.id': 'market-data-producer-1', 'acks': 'all' }) producer.init_transactions() def publish_trade_batch(trades: list[Trade]): producer.begin_transaction() try: for trade in trades: producer.produce( topic=f'market.trades.{trade.exchange}.{trade.symbol}', key=trade.symbol.encode(), value=serialize(trade) ) producer.commit_transaction() except Exception as e: producer.abort_transaction() raise Why Kafka Is the Standard for Market Data Replication
Apache Kafka provides all necessary properties: durability (data stored on disk), scalability (horizontal partitioning), and independence of consumer groups. We configure topics with names like {data_type}.{exchange}.{symbol}.{interval}, making it easy to filter data.
Topic: market.trades.binance.BTCUSDT Partition 0: trades (all, ordered by time) Topic: market.orderbook.binance.BTCUSDT Partition 0: snapshots + diffs (ordered by update_id) Topic: market.candles.binance.BTCUSDT.1m Partition 0: 1-minute OHLCV (ordered by candle time) Delivery Guarantees and Cross-Datacenter Replication
In market data systems, at-least-once delivery is most common: better to get a duplicate than lose data. Consumers are idempotent — deduplication by trade_id or update_id. For risk management and position accounting, we enable exactly-once via Kafka Transactions.
Kafka MirrorMaker 2 replicates topics between clusters. Example MirrorMaker 2 configuration:
# mirrormaker2.properties clusters = us-east, eu-west us-east.bootstrap.servers = kafka-us:9092 eu-west.bootstrap.servers = kafka-eu:9092 us-east->eu-west.enabled = true us-east->eu-west.topics = market\.* us-east->eu-west.replication.factor = 2 The EU cluster receives a replica of all market.* topics with a delay of 50–200 ms for transatlantic replication. This is sufficient for most analytics and risk systems.
Retention Management and Monitoring
Market data accumulates quickly. Retention policies:
- For tick data: 7 days, then delete.
- For daily OHLCV: infinite, with a size limit of 10 GB per partition.
- For order book: log compaction — keep only the latest state per price level.
zstd compression reduces data by 40–70% without noticeable CPU load. Key monitoring metrics:
| Metric | What It Shows |
|---|---|
| Consumer lag | How far consumers are behind the producer |
| Replication latency | Delay between primary and replica clusters |
| Producer send rate | Publication speed (messages/sec) |
| Bytes in/out rate | Throughput |
| Under-replicated partitions | Partitions with insufficient replication |
Consumer lag > 5 minutes for a trading bot is a critical alert. For an analytics system, it's a warning.
Schema Registry and Format Compatibility
To avoid breaking consumers when the schema evolves, we use Confluent Schema Registry and Avro. New optional fields with default null are backward-compatible changes.
{ "type": "record", "name": "Trade", "namespace": "com.exchange.market", "fields": [ {"name": "exchange", "type": "string"}, {"name": "symbol", "type": "string"}, {"name": "timestamp", "type": "long"}, {"name": "price", "type": {"type": "bytes", "logicalType": "decimal", "precision": 24, "scale": 8}}, {"name": "quantity", "type": {"type": "bytes", "logicalType": "decimal", "precision": 24, "scale": 8}}, {"name": "side", "type": {"type": "enum", "name": "Side", "symbols": ["BUY", "SELL"]}}, {"name": "is_maker", "type": ["null", "boolean"], "default": null} ] } What's Included in the Work
- Architectural documentation: topology description, data flow diagram, topic specification.
- Infrastructure access: setup of Kafka clusters, MirrorMaker, Schema Registry.
- Team training: workshop on operations and monitoring.
- Post-launch support: assistance during the first 2 weeks of production operation.
How We Deploy Replication: Step-by-Step Plan
- Requirements analysis: data volume (up to 100,000 messages/s), latency, delivery guarantees, number of data centers.
- Topology design: choose between Hub-and-Spoke, Chain, or Pub-Sub.
- Deploy Kafka cluster (3 to 7 brokers) with MirrorMaker 2 and Schema Registry.
- Configure topics and retention policies.
- Integrate consumers with idempotency and deduplication.
- Set up monitoring (consumer lag, latency) and alerts.
- Test with synthetic data and real loads.
- Document and hand over to the client's team.
During the process, we verify: producer idempotency, transaction settings, retention consistency across clusters, deduplication functionality, and absence of duplicates.
Timelines and Cost
A basic implementation takes 7 to 14 days: analytics, cluster deployment, MirrorMaker 2 and Schema Registry setup, monitoring. A full solution integrated into your infrastructure takes up to 4 weeks. Cost is calculated individually: depends on data volume, number of data centers, and required delivery guarantees. Contact us to evaluate your project and get a consultation. Order the development of a replication system for your tasks — and we will ensure reliable data delivery.