Imagine your DeFi service loses connection to Ethereum for 10 minutes due to a failure—liquidity losses reach $50,000 per hour. Or if it's a payment gateway for a crypto merchant, each hour of downtime costs up to $10,000. A single node is a single point of failure. Node downtime equals product downtime. We configure fault-tolerant blockchain node configurations for production services. Our engineers have experience with Ethereum, Solana, Polygon, Arbitrum, and other networks, over 5 years in the market, and 10+ node configuration projects. The Ethereum documentation states: health check must verify not only connectivity but also synchronization status.
Single Node Problems
Downtime occurs not only from hardware failures. Most often, problems are related to the node falling behind the chain tip—after a crash, Ethereum requires re-synchronization, Solana catches up on slots. RPC overload—a single instance cannot handle request load from multiple services. Planned client updates during rolling updates make the node temporarily unavailable. Hardware failures of disk, RAM, or network card take it offline. Snapshot corruption from an unexpected power outage also leads to lengthy recovery.
Why Choose Active-Active Architecture?
Active-Active architecture solves these problems at the root. It provides 2x faster failover compared to Active-Passive, because both nodes constantly serve traffic. With Active-Passive, the second node idles, and failover requires promotion time. Active-Active instantly redistributes load, and a health check every 5 seconds ensures an unsynchronized node does not receive traffic.
Client requests │ ┌───▼───┐ │ HAProxy / Nginx │ ← health check every 5s └───┬───┘ │ ┌────┴────┐ ▼ ▼ Node-1 Node-2 ← different AZ / datacenters │ │ └────┬────┘ │ Shared or independent storage Example HAProxy configuration for Ethereum RPC:
global maxconn 50000 log stdout format raw daemon defaults mode http timeout connect 5s timeout client 60s timeout server 60s option http-server-close option forwardfor frontend ethereum_rpc bind *:8545 bind *:8546 # WebSocket default_backend ethereum_nodes backend ethereum_nodes balance leastconn option httpchk POST / HTTP/1.1\r\nHost:\ localhost\r\nContent-Type:\ application/json\r\nContent-Length:\ 68\r\n\r\n{\"jsonrpc\":\"2.0\",\"method\":\"eth_syncing\",\"params\":[],\"id\":1} http-check expect string '"result":false' # node in sync if eth_syncing = false server node1 10.0.1.10:8545 check inter 5s fall 2 rise 3 server node2 10.0.1.11:8545 check inter 5s fall 2 rise 3 # Sticky sessions for WebSocket (cannot switch mid-subscription) stick-table type ip size 100k expire 30m stick on src frontend ethereum_ws bind *:8546 default_backend ethereum_ws_nodes backend ethereum_ws_nodes balance source # WebSocket — sticky by source IP server node1 10.0.1.10:8546 check inter 10s fall 2 rise 3 server node2 10.0.1.11:8546 check inter 10s fall 2 rise 3 Critical point for WebSocket: subscriptions (eth_subscribe, Solana slotSubscribe) are stateful; on failover, the client must recreate subscriptions. We use sticky sessions by IP.
What Health Check Does an Ethereum RPC Need?
An HTTP health check (status 200) is insufficient—a node may respond but be 1000 blocks behind. The correct check:
#!/bin/bash # /etc/haproxy/scripts/check_eth_node.sh NODE_URL="http://localhost:8545" # 1. Ensure node is not in sync process SYNCING=$(curl -sf -X POST "$NODE_URL" \ -H "Content-Type: application/json" \ -d '{"jsonrpc":"2.0","method":"eth_syncing","params":[],"id":1}' | \ jq -r '.result') if [ "$SYNCING" != "false" ]; then exit 1 fi # 2. Ensure last block is not older than 3 minutes (180 seconds) BLOCK_HEX=$(curl -sf -X POST "$NODE_URL" \ -H "Content-Type: application/json" \ -d '{"jsonrpc":"2.0","method":"eth_getBlockByNumber","params":["latest",false],"id":1}' | \ jq -r '.result.timestamp') BLOCK_TIME=$((16#${BLOCK_HEX#0x})) NOW=$(date +%s) AGE=$((NOW - BLOCK_TIME)) if [ $AGE -gt 180 ]; then exit 1 fi exit 0 Similarly for Solana—check getSlot and getEpochInfo, tolerance 50–100 slots.
| Health Check Method | What It Checks | Drawback |
|---|---|---|
| HTTP 200 | Port availability | Does not detect lag |
| eth_syncing | Synchronization | Does not give block age |
| eth_getBlockByNumber | Last block timestamp | Requires script |
How to Update Nodes Without Downtime?
Rolling update—update one node at a time, removing it from rotation:
#!/bin/bash # rolling_update.sh # Step 1: remove node1 from rotation haproxy -sf $(cat /var/run/haproxy.pid) -f /etc/haproxy/haproxy_node2_only.cfg # Step 2: wait for drain of existing connections sleep 30 # Step 3: update node1 ssh node1 "systemctl stop geth && apt upgrade -y ethereum && systemctl start geth" # Step 4: wait for node1 sync while ! /etc/haproxy/scripts/check_eth_node.sh node1; do echo "Waiting for node1 to sync..." sleep 30 done # Step 5: return node1, update node2 haproxy -sf $(cat /var/run/haproxy.pid) -f /etc/haproxy/haproxy.cfg sleep 30 ssh node2 "systemctl stop geth && apt upgrade -y ethereum && systemctl start geth" How to Monitor HA Cluster State?
Use Prometheus + Grafana with metrics:
| Metric | Alert Threshold | Severity |
|---|---|---|
| eth_block_age_seconds | > 120s | Critical |
| haproxy_backend_active_servers | < 1 | Critical |
| haproxy_backend_response_time_ms | > 2000ms | Warning |
| node_disk_io_time_percent | > 80% | Warning |
| node_memory_available_bytes | < 10% | Warning |
Set up alerts in PagerDuty or Telegram. For backend_active_servers < 1—immediate notification to the on-call engineer.
Common Mistakes When Setting Up HA
When configuring HA, common mistakes include: identical health checks for all nodes, lack of block age monitoring, ignoring WebSocket sticky sessions, manual updates, and insufficient capacity for resync. Let's break down each case. If you only check the port, the load balancer will consider both nodes healthy even if one is lagging—requests get outdated data. A node could hang at block 50 for hours, and the health check won't trigger—use eth_getBlockByNumber with timestamp. A client subscribed to events via one node loses subscriptions on failover without reconnection. Without a rolling update script, downtime is easily introduced—automate with HAProxy and API. After a crash, a node may re-sync for hours—account for this when choosing disks and bandwidth.
Process: From Analysis to Deployment
- Analysis—Study current infrastructure, uptime requirements, budget.
- Design—Choose scheme (Active-Active or Active-Passive), providers, monitoring tools.
- Implementation—Deploy nodes in different availability zones, configure load balancer and health checks.
- Testing—Simulate failures, verify failover and recovery time.
- Deployment and Documentation—Deliver configuration, maintenance instructions, and action plans for failures.
What's Included in the Service
- Deployment of a second node in a separate AZ/datacenter
- Configuration of HAProxy or Nginx with smart health checks
- Rolling update scripts for zero-downtime updates
- Prometheus metrics, Grafana dashboard, alerts
- Documentation of failover and recovery procedures
If your system requires 99.9% uptime—contact us for an audit of your current architecture. Order fault-tolerance configuration for your blockchain node—we will find the optimal solution within 5 business days. Get a consultation: we'll evaluate your project and propose a plan.