REST and WebSocket API Development for Crypto Data

Why a REST API for crypto data cannot handle market load

The access layer for market data is designed first, and it breaks first. A REST API for crypto data works on a request-response scheme: the application goes to the source for every quote and waits for a response.

While the market is calm, the scheme looks workable—latency is predictable, and the number of calls grows linearly with the number of users.

At peak volatility, the picture changes abruptly. Request limits to the source are exhausted in seconds, some calls are rejected, and timeouts under peak volatility become the norm rather than a rare exception.

The client goes into retries, adds load to the access layer, and closes the loop: the more retries, the fewer successful responses reach the product.

Then stale quotes appear in the response. If the access layer returns a cached value without marking its age, the application shows a price that no longer exists on the market—and for trading or settlements this is worse than an honest error.

Desync is added: some instruments come through one channel, some through another, their freshness differs, and consolidating everything into a single market snapshot becomes a separate task.

Ultimately, load on the access layer grows not from the number of users but from the very architecture of source access: adding servers does not remove the problem, because the bottleneck is the external limits of the source. We explain how this is solved.

What changes after rebuilding the access layer for blockchain data

Response speed and stability under peak load

Rebuilding the access layer begins with separating flows: market data—prices, order books, trades—is read from a cache on top of Redis with a short TTL, while requests to the blockchain network go through a queue with a limit on concurrent calls to the node.

Caching market data removes most retries, so a spike during high volatility does not turn into a queue of waiting clients.

In our measurements in production environments, response time for frequent requests decreased severalfold, and the error rate under peak load stays within tenths of a percent.

A separate class of failures due to external source unavailability is also addressed: when a provider fails, the request is served from the last reliable snapshot, and subscriptions reconnect themselves. Recovery after a source failure takes minutes instead of hours of downtime.

How much manual work is removed from processes

Previously, the resilience of the access layer depended on a person: they monitored node lag, restarted data streams, and manually reconciled discrepancies between the source and their own database.

Now these operations are defined by rules—lag control, exponential backoff retries, automatic discrepancy reconciliation, and alerts only for anomalies.

The reduction in manual operations is visible in on-call duty: hours of routine checks per week are eliminated. The access scheme, caching rules, and recovery procedures remain in the documentation, so the result is not tied to a specific engineer.

Data delivery modes: polling on demand, streaming subscriptions, hybrid

The delivery format is chosen based on how often the data changes and how many clients read it simultaneously.

For trade history, period aggregates, and reports, polling on demand gives predictable load: the client sets the window and frequency, and the failure of one call does not affect the others.

For quotes, order book, and transaction statuses, streaming subscriptions are needed—the server sends the change immediately, without a waiting period.

When polling on demand is cheaper to support

Polling on demand wins where data changes less often than once every few seconds: there is no long-lived connection state, no resubscription after a disconnect, and debugging uses regular call logs.

Responses are cached on the service side, so a peak of requests from multiple clients does not become a peak of calls to the source. The comparison of data retrieval modes here is clearly in favor of maintenance simplicity.

Hybrid scheme: hot data in a stream, history on demand

The hybrid delivery scheme removes the main limitation of subscriptions: only what changes now remains in the stream, while depth and history go to requests with paginated output.

After reconnection, the client backfills the missed segment with one call instead of rereading the entire channel. This keeps the interface responsive and storage load predictable.

Work order: request audit, schema, configuration, verification, handover

We proceed through the stages so that each next step relies on recorded data from the previous one, not on assumptions.

First we analyze actual calls to sources, then design the schema, and only then configure limits and queues—otherwise parameters are chosen at random and fail at the first peak load.

1. Request audit: capture the request log to sources for a working period—who, what, and how often requests, where duplicates and idle calls are.
2. Source integration scheme: for each data type, define the primary and backup source, normalization format, and priority for value discrepancies.
3. Configuring limits and queues: distribute quotas among consumers, set spike buffering and a degradation rule for when the source responds slowly or returns errors.
4. Load testing: run a profile close to peak, measure response latency and failure rate, and verify that queues drain without latency growth.
5. Handover of the environment: deliver configurations, access, schema, and incident response procedures, and review typical incidents with your team.

What we check during the audit

Measurements matter here, not general words: we record the frequency and request profile for each source, its limits and actual response latency, and note failure points—a source without backup, a hard limit without headroom, no retries on connection loss.

Using this data, we calculate quota headroom and decide where buffering is needed and where a local cache is sufficient.

What remains with the customer's team: configurations, schemas, regulations

What is handed over is not a set of credentials and verbal agreements, but a set of artifacts your team can use to bring up the service, connect a new source, and investigate a data discrepancy without our help.

Composition of transferred documentation

• Service configurations: environment parameters, request limits, subscription reconnection rules, and timeouts.
• Data flow diagrams: the path from source through normalization and cache to the delivery point, indicating possible loss and duplicate points.
• Access matrix: which services and roles are authorized for which methods, keys, and streams, with read and administration separation.
• Interface documentation: description of REST methods and WebSocket channels, error formats, limits, request and response examples.
• Data model: unified format for instrument, pair, trade, and candle, and rules for reconciling data from multiple sources.
• Test environment description: how to bring up a test bench and reproduce an incident on historical data.

Regulations and areas of responsibility

Separately, the source update procedure is transferred: the procedure for adding a new source, compatibility check of the format in the test environment, and conditions for inclusion in production.

The same document records the responsibility boundary: what our on-call shift monitors during the support period, what transfers to yours, how response schema changes are documented, and where to look when quotes diverge between sources.

Implementation example: exchange aggregator — growth in requests without failures

The client is an exchange aggregator that consolidates quotes and order books from dozens of venues and delivers them to external applications via a public REST API and WebSocket stream.

The system was unprepared for audience growth: attempts to scale it by adding new API methods on top of a shared monolith only worsened the problem.

What it was like before the rebuild

One service combined market data collection, calculations, and client delivery, so peak traffic hit all nodes at once.

The failure rate stayed at 9-14% during high activity hours, response time on heavy queries reached several seconds, and operators edited limits and cache manually—up to twenty interventions per week. Incident investigation took hours: it was not clear at which segment the data was lost.

Result in measurable indicators

Collection and calculations were moved to a separate environment, and client delivery to its own, with independent scaling and cache for hot subscriptions.

The failure rate at peak is 0.2–0.4%, response time under load is tenths of a second with peak load tripled, and manual edits are no more than one or two per month.

The metrics were taken from operational dashboards in Grafana; they remained with the customer along with the alert settings.

When a single venue degrades, the subscription does not break: the client receives data from the backup channel, and the event goes to the log for investigation. This margin of resilience made it possible to grow the audience without revisiting the architecture.

What to do if your own node and API are already deployed

A deployed own node and a working API are not a reason to rewrite the environment entirely. First we take inventory: which methods generate traffic, which consumers depend on them, where requests go directly to the node, and at what volume it starts lagging behind the network. This gives a decision map—what stays as is and what moves under the new layer.

The key change is cache and queue between consumers and the node. Hot data such as balances and latest blocks are served from cache with a configurable TTL, heavy historical queries go to the queue and are computed in the background.

Event subscriptions are handled by a separate channel, so an activity spike does not bring down the main requests, and the node does not duplicate the same work hundreds of times.

Zero-downtime migration is built on gradual transfer. The new layer is brought up alongside the old one, a share of traffic is transferred to it, and responses are compared with the previous ones on live requests.

If they match, the share grows; if they diverge, the switch is rolled back with one command, and consumers do not notice the difference.

Existing integrations are preserved: method addresses and response format remain the same, only the internal implementation changes.

Phased rollout proceeds one method at a time, so the product never goes without data for a minute, and each switched part is immediately visible in the metrics.

Technical doubts: load, source failures, support

Before the start, customer engineers ask the same set of questions: how the environment behaves when a source fails, how to make sure data is not lost in transit, how long connection takes, and what happens after launch.

Below are the answers in the form we give them in technical sessions, based on operating similar environments: the service does not trust the source and checks itself.

How do you verify data completeness?

Continuity of the event sequence, reconciliation of aggregates with an independent source (for example, CoinGecko), and the share of missed intervals. If the gap exceeds the threshold, the window is marked unreliable and does not go to the consumer silently.

How long does onboarding take and what about support?

The first environment on one source takes days: adapter, data schema, checks, and a run on recorded traffic. After that—on-call duty based on latency and error rate metrics, adapter updates when the response format changes, and scheduled recovery checks from the log.

How do you add or replace a source later?

The adapter is separated from the core: a new provider is connected with its own priority and reconciliation rules. Replacement is canary—some traffic to the new source, metric comparison, then switching the primary.

Start of work: what we need from you and how we will return with an estimate

To return with an estimate and work plan rather than general assumptions, we need a minimum of inputs. Some are about the data itself, some about access and constraints on your side.

• List of sources: exchanges, brokerage systems, or own services from which we take quotes, order book, trades, and history.
• Where periodic requests are sufficient and where a persistent WebSocket stream with reconnection and missed-data backfill is needed.
• Source keys in read-only mode, without rights to trading operations.
• Access to the deployment environment or repository and an understanding of where to write logs and metrics.
• Data provider limits: request frequency, history depth, behavior when exceeded.
• Requirements for access separation and request logging.
• Contact on your side who will promptly confirm field format and answer clarifications.

The start order is as follows: we discuss the task, record the scope of sources and consumption scenarios, then return with an estimate and work plan—stages, timelines, and acceptance points.

The first stage of work removes the main risk and usually goes faster than the rest: describe endpoints and stream format, bring up a REST API skeleton for crypto data on one source, and test behavior on connection loss, request retry, and data discrepancy between sources. Then we connect the remaining sources using the already proven scheme.

Describe the task—what data is needed and who will consume it; we will clarify the rest in conversation.