Why a go-ethereum indexer lags behind the chain head
The node is synchronized with the network, while the data layer for the application lags behind.
A blockchain indexer reads blocks, spreads them across tables, and serves them to the application — and it is precisely at this junction that lag behind the chain head appears: the interface shows a balance that no longer exists, or misses an event confirmed several minutes ago.
The causes are usually engineering-related, not “the network is slow.” Batch writing to the database with a wide window, sequential processing in a single thread, subscription to new blocks without reconnecting — after a disconnect, the range is lost, and the indexer on go-ethereum continues from the last written number, leaving a gap in blocks.
A reorg at that moment rewrites the canonical branch, while the application has already served data for the previous one.
The cost of such lag is state discrepancy.
A DeFi protocol’s calculation relies on an outdated balance, an NFT showcase misses a minted token or a change of owner, and a support request can be reproduced only on the user’s side: in the database and in the API everything is “correct,” just from the past.
Lag does not resolve itself: the missed range must be caught up, and the rewritten history must be correctly overwritten without duplicating events. Below, we break down how this is closed: from confirmations and idempotent writing to a separate cursor and a lag metric in blocks.
What a reliable data layer provides for DeFi and NFT services
Everything the user sees passes through the data layer: balances, transaction history, pool position state, token owners.
If this is collected directly from the node on every request, update lag grows with the number of clients, and desynchronization at the moment of transaction signing serves outdated numbers to the user.
We build such a layer on go-ethereum — behavior during reorganizations and repeated block processing is predictable, because the client already solves these tasks itself.
Freshness of balances and events
The indexer advances the state pointer only after a block is confirmed at a given depth of confirmations.
This removes an entire class of failures — “showed a balance that no longer exists”: before confirmation, data resides in an intermediate layer and does not reach the interface.
During a chain reorganization, recalculation follows the event log — the service returns to a consistent state within minutes instead of rereading history from scratch.
Streaming updates are delivered over WebSocket: the subscriber receives a delta for its addresses instead of polling the node, so update lag fits within the interval of a single block.
Read resilience under peak load
Reads are separated from writes: the indexer writes in batches to storage, while the API and subscriptions read from replicas and a hot cache.
Peak activity comes from growth in the number of addresses, not from the speed of a single node — read load is distributed horizontally, and response time remains stable when a spike adds subscriptions.
Failures degrade partially: if a replica is unavailable, requests go to the remaining ones, and the indexer continues writing. The data layer removes two classes of problems from the team at once — unreliable balances and read unavailability at the most inconvenient moment.
Indexer architecture options on go-ethereum
The indexer design determines service behavior when the source fails: whether the API will continue serving balances and events or will error out. Below are four configurations that we use on go-ethereum-compatible networks and their real limitations.
| Option | When it fits | Limitations |
|---|---|---|
| Own full node with an indexer | Private tracing methods, access to the transaction pool, predictable latency | Disk and initial sync grow with the network; client update monitoring is required |
| External network access provider | Quick start, infrequent requests, several networks at once | Rate and history depth limits, no private methods, dependence on someone else's availability |
| Hybrid: own node as primary, external as backup | A mode where a pause in indexing is unacceptable | Harder maintenance: two sources, lag monitoring, failover |
| Streaming subscription over WebSocket | Online notifications and lightweight services without history | A missed block cannot be restored; state is lost on restart |
When your own node is better than an external one
We take an own node when history is loaded in batches and the external provider cuts query depth and request rate. Private tracing methods and transaction pool reads are often unavailable externally; locally, these are ordinary calls to your own port.
Response latency does not depend on someone else's queue, and we update the client version on our own schedule.
Hybrid mode and source redundancy
We build the hybrid like this: the node is the primary source, the external provider is the backup. The indexer writes to its own database and idempotently replays block ranges, so switching does not break the data.
A watchdog process compares the last processed block with the network and, if lag exceeds the threshold, moves traffic to the backup, then returns it after catch-up.
Implementation stages: from node audit to handover
Audit and design come before the first line of code: catching an error in the data flow design here is cheaper than redoing it after load. Each step ends with an artifact that can be checked before moving to the next. Below is how the work is structured.
- Node and data flow audit. We capture the current state: which clients are used, where events are lost, how subscriptions are structured, and with what lag data reaches the interfaces. We record bottlenecks and discrepancies between the network and the database.
- Design of the indexing and storage schema. We determine which events we read, how we link them to state, and which indexes are needed for typical wallet, pool, and collection queries. The schema is agreed upon before implementation.
- Chain rollback handling. We set the confirmation depth, the order of block replay and cleanup of affected records — after a reorganization, no nonexistent balances or owners remain in the database.
- Deployment of read interfaces. We bring up the query interface and streaming subscriptions, limit request rate, and enable caching on hot queries.
- Load testing. We run a profile of real traffic: peak request rate, mass reindexing, indexer lag, and behavior when the node is unavailable.
- Handover to operations. We hand over configurations, schemas, restart and recovery policies, and instructions for the on-call shift.
Checkpoints at each step
Each stage has a measurable criterion: indexing lag, the share of correctly replayed blocks after a rollback, response time for a hot query, recovery without manual edits.
Until the criterion is met, we do not move on — data discrepancy is detected during setup, not in production.
What we hand over: schemas, policies, procedures
The deliverable is not a source archive, but a set of artifacts that your team can use to deploy, roll back, and maintain the services without us. Below is exactly what is handed over and what task each document covers.
- Service configurations — node connection (in the baseline setup — go-ethereum), concurrent request limits, batch size for block reads, timeouts, and retry policy. For each value, behavior under growing load is specified.
- Deployment and topology diagrams — which components are stateless, where the database and queue live, how traffic flows from the balancer to the node and socket subscribers.
- Access policies and roles — separate accounts for services, the on-call engineer, and analytics; least-privilege rights; key rotation procedure.
- Recovery procedure — target recovery time and recovery point, restore from snapshot, block replay after a discrepancy, and reconciliation with the network.
- Data format description — table and index schemas, socket message format, idempotency rules, and chain reorganization handling rules.
- On-call shift instructions — which alerts are considered routine, the diagnostic tree, what the on-call engineer can safely restart, and what requires calling an engineer.
Documentation for the customer's engineering team
A separate block covers what is needed for development, not operations: extension points in the code, records of key decisions and the reasons for them, an acceptance checklist, and the procedure for connecting new contracts and metrics.
It also defines the boundary of responsibility: what remains with us and what is handed over to your team.
Case study: balance discrepancies reduced to zero
A DeFi protocol with liquidity pools and an NFT showcase came to us: a Go backend read the chain via go-ethereum, stored state in a relational database, and served balances via REST API and WebSocket.
The user saw one amount in the interface, while the contract confirmed another — support was drowning in requests about “stuck” balances.
Before the rework, the indexer processed blocks sequentially: lag behind the chain head was 40–90 seconds, the discrepancy rate in reconciliation was up to 0.8% of addresses, and recovery after a crash took more than 40 minutes because the process started from scratch.
At peak — a fivefold surge in operations — the block queue grew, event subscriptions dropped, and some transfers were not reflected at all.
What exactly was changed in the indexing schema
The bottleneck was not requests to the node, but state writing and the lack of resilience to reorgs.
Blocks began to be collected into batches and processed in parallel; writing a range happens in a single transaction; logs are written idempotently by the key “block + log index,” so a repeated pass does not cause double crediting.
Reorgs are handled by a separate circuit with rollback by checkpoints instead of a full rebuild; WebSocket sends only the balance delta, and a daily reconciliation with a contract call highlights discrepancies earlier than the user.
After implementation, lag stays within 1–2 seconds, there are no discrepancies across 12 thousand address checks, and recovery after a crash takes 3–4 minutes — the process resumes from the last checkpoint.
A fivefold peak in activity does not grow the queue: batches are written in time, and subscriptions do not drop.
How does an archive node differ from a full node?
The difference between node modes lies in which part of the network state it stores and at what history depth it is ready to respond.
A full node keeps the current state and recent blocks: fresh requests are served quickly, but a request for an arbitrary past block will return an error.
An archive node stores intermediate states from the very beginning of the chain — history depth is maximal, but storage volume and initial sync time grow.
A light client does not store state at all: it receives block headers and verifies data using proofs. This is enough for a wallet, but not for a backend that builds indexes and recalculates historical queries.
The mode is set at client startup: in go-ethereum, these are different sets of state sync flags, and switching between them means reloading data, not changing a setting on the fly. Therefore, the node mode is chosen before deployment — based on which requests the service handles.
| Mode | History depth | Response speed | Storage | Scenarios |
|---|---|---|---|---|
| Full node | current state and recent blocks | high on fresh data | moderate | processing new events, subscriptions, current balances |
| Archive node | the entire chain from the first block | high on any block | multiple times larger | audit, historical queries, full index recalculation from scratch |
| Light client | headers only | depends on an external node | minimal | wallets, lightweight transaction verification |
Technical concerns before project start
Before estimating timelines, engineers check four things: where the node lives, what happens to the service during an update, how the system behaves under growing load, and who is responsible for data discrepancies.
Below are short answers; we put the responsibility boundary and acceptance criteria in writing before the start.
We already have a node deployed — do we need a second one?
No, we connect to the existing one in archive access mode. If it does not store the full state history, the go-ethereum indexer will calculate the missing blocks itself, without resynchronizing from scratch.
Will the update stop the service?
No. The new circuit is brought up alongside, catches up to the current block height, and only then is traffic switched. The old one remains operational until the first reconciliations — rollback takes minutes.
What happens during a sharp increase in activity?
Blocks are read in batches, writing is a single transaction, so processing time grows linearly. The API and WebSocket are separated into dedicated processes: a spike in subscriptions does not slow down event writing.
How is data correctness verified?
Each block is verified by number and hash, and address state is periodically recalculated from checkpoints. A discrepancy stops indexing progress and goes into a report rather than quietly into API responses.
Let's discuss the task and come back with an estimate and plan
A detailed document is not needed to start the discussion — what your team already has is enough. Send the usage scenario: which operations are read and written, which data is needed in real time, and which can be collected in batches.
If you have a data schema, contract models, and the current node code on go-ethereum — add them: they show where the bottleneck is and what will have to be rewritten.
We break down the task in two steps: a call on architecture, then a review of the data schema and failure scenarios — what we do during a chain reorganization, how we restore the index after a failure, and where the boundary between the node and the service layer lies.
Based on the results, we propose a pilot run on a test network: it shows the real sync speed and removes the main risks before the main work begins.
In response, you will receive:
- scope of work by stage and responsibility boundaries;
- data schema: tables, indexes, reindexing rules for reorgs;
- API contracts and WebSocket message format with examples;
- pilot launch plan: scope, duration, success criteria;
- failure scenarios and index recovery procedure;
- timeline estimates by stage and environment requirements;
- acceptance criteria and documentation handover procedure.
Describe the task in free form — we will come back with an estimate and work plan.