Scaling blockchain infrastructure: where it breaks under load
Under load, infrastructure breaks not where it is usually sought. The first to give way is not the disk or CPU, but the request queue: a JSON-RPC node accepts them into a limited buffer, and at peak activity the queue grows faster than it is drained.
Then node response degradation kicks in — read response time grows nonlinearly because requests compete for the same states.
The second bottleneck of a blockchain network is desync. One node advances in height, another lags several blocks, and the balance read from the lagging one diverges from what the client sees.
During reorgs this is more noticeable: the cache returns stale state, and the index built from logs lags the chain head.
The third is state and subscriptions. A request for historical data on an archive node reads thousands of keys, and such a call holds the thread while others wait.
Persistent WebSocket subscriptions copy the load differently: each new listener does not add users, but adds connections that must be kept open and reconnected on disconnect.
Each of these points has a limit beyond which response drops in steps, not smoothly.
Node load grows more from the number of subscriptions than from the number of users — therefore scaling blockchain infrastructure starts with analyzing where exactly the buffer ends and what happens after. We break down how this is addressed.
What network restructuring gives: speed and fault tolerance
A single node that simultaneously serves history, accepts transactions, and broadcasts events is the most common bottleneck.
As load grows, node throughput hits a single process: read response time increases faster than the number of clients, and the event stream starts lagging the network state.
So we separate roles — validating, archive, and request-serving — and scale each group separately according to its load profile.
Fault tolerance of a blockchain network is not “a node in two instances.” We keep standby hot, with one-block lag, and the load balancer checks liveness of each node and removes traffic from one that stopped responding. Failover takes seconds, and service downtime is measured not in minutes but in switchover time.
Subscriptions are a separate story. A WebSocket connection is tied to a specific node, so when it fails, the client silently loses the event stream.
We keep node-bound sessions and a buffer with a cursor of the last processed block: after a disconnect, the client catches up on missed events rather than starting from scratch. The share of lost subscriptions goes to zero.
On Ethereum and compatible networks, such a layer gives predictable metrics: read response time stays within limits set by the load profile, recovery after node failure fits into seconds, and the failure class “service unavailable due to a single node” disappears from statistics.
What exactly we change in a specific network is determined by measurement before restructuring, not by a generic template.
Modes: vertical scaling, distribution, hybrid scheme
The three modes differ not in power reserve but in where the limit arises. Vertical scaling is a more powerful node: state and the pending transaction queue entirely on one machine, so response to requests is predictable and consistency logic does not get more complex.
You pay with the hardware ceiling and long synchronization: full reindexing takes hours, and any maintenance is a downtime window.
Horizontal node distribution is built differently: several replicas behind a load balancer, reads and subscriptions go to them, writes and signing remain on the primary node.
A hybrid scheme combines both approaches — a reinforced node for writes plus a pool of replicas for read traffic and WebSocket subscriptions.
| Mode | When it fits | What you pay with |
|---|---|---|
| Vertical scaling | Load grows moderately, the number of clients is predictable, a simple scheme matters | Hardware ceiling, hours for synchronization, downtime during maintenance, accelerating growth in index size |
| Horizontal node distribution | Many parallel reads and subscriptions, requests allow slight data lag | Replicas lag: some responses return stale state, lag control and careful client migration between nodes are needed |
| Hybrid scheme | Mixed profile: heavy writes and mass reads at the same time | More expensive to maintain: two contours, separate update and fault-tolerance rules, more failure points |
The choice is determined not by the “most powerful” option but by the one whose limit lies further than your load profile. We calculate read share, number of concurrent subscriptions, and data freshness requirements — and only after that fix the node operating mode.
Work order: from assessment to handover into operation
Scaling does not start with deployment but with measuring current limits: how many concurrent subscriptions the gateway holds, at what call volume the data read layer hits its limit, how synchronization behaves when a node lags. Below is the work order, where each step ends with a result accepted before moving to the next.
Seven steps with measurable result
- Infrastructure assessment. We capture the load profile: concurrent subscriptions, composition and frequency of calls, peak intervals, disk and network headroom. The output is a report with actual limits, not eyeball estimates.
- Node placement scheme. We distribute roles — validating, read nodes, archive nodes, subscription gateways — and fix connections and failure points. Topology is agreed before configuration.
- Node configuration. Configuration for the captured profile: connection limits, timeouts, storage depth, read/write separation, replicas. Every change is versioned and reversible.
- Load testing. We reproduce peak and headroom above it, check behavior when one node fails: smooth degradation or outage. We adjust configuration based on metrics.
- Pilot operation. Real traffic with extended observation: compare forecast with fact and fine-tune what behaves differently.
- Observability and procedures. Dashboards, alerts, diagnostics and rollback procedures — so a failure is caught before users notice it.
- Handover into operation. Access, recovery instructions, and on-call shift training: the system is raised and repaired without our involvement.
What remains with you: configurations, schemes, procedures
Infrastructure without documentation lives exactly until the first engineer leaves or an incident occurs.
Therefore everything we configure is handed over as working artifacts: they can be checked, redeployed in a clean environment, and given to your team without verbal retelling. Below is the kit that remains with you after handover.
- Node configurations — synchronization parameters, connection limits, timeouts, retry policy, cache sizes; alongside an explanation of why this value was chosen and what changes as load grows.
- Topology schemes — distribution of nodes across zones, backup routes, duplicated sections and points whose failure would bring down the entire contour.
- Access rules — roles, keys, read/write permission boundaries, issuance and revocation procedure, action log.
- Traffic distribution settings — how requests are distributed across nodes and what happens when one of them fails.
- Maintenance procedures — planned work, update procedure, health checks, alert thresholds, and on-call actions.
- Recovery procedures — backups, target recovery time, rollback to previous state, scenario drills.
- Infrastructure documentation — service map, dependencies, owners, instructions for the on-call engineer.
Each item is not a description but a working file or tested scenario: it can be handed to a new team, and they will bring up the contour without us.
Case: threefold load growth without response degradation
Before restructuring, infrastructure relied on a single node with a shared public endpoint.
The application grew, the number of requests tripled over a quarter, and during peak hours subscriptions to new blocks dropped by the dozens: clients saw event gaps, and recovery of a failed node took about forty minutes of manual engineer work.
We started with load testing — reproduced a threefold peak on a test bench and found the failure point: it was not the network that overflowed but the subscription queue inside the node process.
Then we separated roles: reads went to a pool of nodes behind a load balancer, writes and event broadcasting — to a separate subscription layer with buffering and retry, and orchestration on Kubernetes replaced manual node startup with automatic restart with health checks.
Metrics before and after
- request volume — from 1.2 to 3.6 thousand per minute on the same machine fleet;
- response under peak — 95th percentile reduced from 4 to 1.1 seconds;
- subscription drops — from 60–80 per peak hour to zero over a month of observations;
- node recovery — from 40 minutes manually to 90 seconds automatically.
Load headroom remained: the next threefold growth will hit not nodes but event history storage — this is the next point we keep under observation.
We separately established a rollback procedure: the subscription layer is deployed separately from nodes, so an unsuccessful update is rolled back without losing the event stream.
What to do if blockchain infrastructure is already deployed?
Already deployed nodes rarely require full service shutdown — downtime occurs where there is a single copy and one path to data. First, inventory of blockchain infrastructure: which instances serve reads and writes, where the load balancer is, who holds event subscriptions. Then a parallel node is raised nearby and synchronized with the network.
Node migration goes through reconciliation, not a head-on switch. The new instance is compared with the old on control block ranges — a state discrepancy stops the work until the cause is determined.
If it matches: the node is enabled as a read-only replica, and its network lag and error response share are monitored.
Everything above the storage layer changes on the fly: RPC endpoints, load balancer, quotas, response cache, read replica set. The indexer is rebuilt in parallel — the new one writes its database, the old one continues serving data. Event subscriptions are migrated with overlap so the client does not lose a block at the junction.
A short maintenance window is needed where the storage format or node role in the network changes. Database schema change, migration of the primary write node, validator key rotation, and preparation for a network consensus upgrade are synchronized by block height. During this time the service is switched to replicas: reads are not affected, only writes stop.
The window is chosen based on minimum network activity, and on switchover there is always a way back: the old node is shut down not immediately, but after the new one has run without errors for the entire observation period.
Thus modernization without downtime becomes a procedure — a clear order, verifiable points, and predictable behavior during failure.
Common technical doubts before starting work
Difficulties in working with nodes are rarely related to code — they manifest at the intersection of load, hardware, and responsibility boundaries. We collected five engineering-level questions most often asked before starting.
What to do if hardware requirements grow after launch?
We allow disk and memory headroom and describe thresholds after which a different machine class is needed. The node and access services scale separately, so the entire scheme does not have to be redone at once.
Who is responsible for failure after node handover?
We hand over configurations, network schemes, and procedures by which the on-call team reproduces any state. We separately define the responsibility boundary: what our monitoring covers and what remains on your side.
How to determine where the bottleneck is — node, network, or client access?
We measure in parts: processing time on the node, channel latency, and access service behavior under load. It shows what to scale — storage, throughput, or connection points.
What happens under peak load and connection drops?
We limit the number of connections per node, add replicas, and reconnect with block catch-up. The client does not lose data, and the node does not go into overload entirely, but within a single replica.
Should several nodes be deployed at once?
We start with one, capture the load profile, and add replicas based on metrics. Redundancy appears when it is actually needed, not “just in case.”
Let’s discuss the task: how a work plan is built
A work plan for scaling blockchain infrastructure is built not on guesses but on measurements: without understanding the current topology, it is easy to optimize the wrong place where throughput is actually lost.
To discuss the task substantively and return with an honest timeline estimate, we need initial data — here is what the conversation consists of.
- Current topology: how many nodes in the contour, what roles they serve (validating, read, archive, indexers), how they are placed and connected to each other.
- Load profile: peak requests per second, share of real-time subscriptions, daily and weekly rhythm, system behavior at peaks.
- What hurts now: node lag from the network, connection drops, disk growth, index desync, failures when serving data.
- Availability requirements: target recovery time after failure, acceptable degradation, whether rollback to previous state is needed.
- Constraints: where nodes are allowed to be hosted, which cloud platforms and jurisdictions are allowed, how access is issued and who is responsible for on-call.
- Acceptance criteria: by which metrics we will consider that things got better — response latency, share of lost subscriptions, reload time after restart.
In response you receive a topology scheme with marked failure points, a phased work plan, a timeline estimate with dependencies, and a sequence of changes in which the service remains available at every step. If there is no data yet, we will advise which measurements to take so the picture becomes complete.
Describe the task in free form: current state, what stopped coping, and what behavior you want to reach. We will return with an estimate, a plan, and questions that should be closed before starting.