Why standard nodes can't handle the data stream
A full node keeps the state of the entire network and serves data through RPC calls — convenient for a wallet, but not for analytics and product scenarios.
Every history request is either a scan through blocks or eth_getLogs over a narrow range, and as soon as a single block contains several thousand events, the node starts returning responses in chunks or drops the connection entirely on a timeout.
The application receives not data but fragments, and a coherent picture can no longer be reconstructed from them.
Indexing lag is the second systemic problem.
The node syncs with the network, but your read layer lives as a separate process and under peak load cannot keep up with the arrival of new blocks: the queue grows, and the lag between an on-chain event and its appearance in your database is measured in tens of minutes. For trading scenarios, payouts, and monitoring, this means decisions made on stale data.
The third class of failures is missed events during chain reorganizations and connection drops.
If the indexer does not track the change of the canonical branch and cannot roll back already written records, events from discarded blocks remain in the database: balances and histories diverge from the actual state of the network, and finding the source of the discrepancy after the fact is nearly impossible.
A standard node is not designed to be the source of truth for a product — it is engineered as a consensus participant.
That is precisely why a custom blockchain indexer is assembled for tasks with a dense stream of events: a dedicated read pipeline, idempotent writes, confirmation depth control, and explicit handling of reorganizations. Let's break down how this is addressed in practice.
What a custom data processing pipeline provides
Indexing latency stops depending on someone else's infrastructure
Public explorers serve data through a shared pool: at peak load, requests for fresh blocks queue up, and the application receives network state with a lag you don't control.
A custom pipeline reads blocks directly from the node and stores them in its own database — the delay between a block being produced and its availability in the API stays within a predictable corridor and doesn't grow with someone else's load.
For operations where state freshness matters, this is the difference between "yesterday's data" and "data from the last few seconds."
History completeness and handling reorganizations
External sources often serve a truncated history: some internal transfers, event logs, and service calls don't make it into the public index.
A custom pipeline writes everything that passed through a block, including internal transactions and contract events — queries by address or token don't lose records at depth.
Reorganizations are handled separately: when the chain rolls back, the indexer marks the affected blocks as invalid and rebuilds the affected range, so no "dangling" states remain in the database that later surface in reports.
Resilience to network upgrades and fewer manual operations
The network changes block formats, adds new event types, updates contracts — a public explorer adapts to this on its own, while your queries break silently.
A custom pipeline describes the data schema explicitly: when the format changes, a single parsing module is fixed, and the rest of the logic and the API stay the same.
This eliminates regular manual rework during network upgrades — the team doesn't spend cycles figuring out "why the field disappeared" and instead gets a stable data stream it can rely on in the product.
Ways to organize indexing: comparing modes
The choice of mode determines not only how quickly data appears in the index, but also the cost of infrastructure, and how the system survives a rebuild of history. In practice, there are two pure schemes, while the working configuration is almost always a third one — hybrid. Below is a comparison of when each mode is appropriate and where it starts to hit limits.
| Mode | When it fits | Limitations |
|---|---|---|
| Streaming indexing | Fresh data with second-level latency is needed: balances, transactions, contract events in real time | Requires a stable connection to the node and a queue; if the connection breaks, gaps are possible, and order control and redelivery of blocks are needed |
| Batch processing | Initial loading of history, analytics, reports where completeness matters, not seconds | Data appears with a delay; over large block ranges, traversal time and load on the node grow |
| Hybrid mode | A product with a live interface and deep history at the same time | Harder to maintain: two loops, state reconciliation, and a single point of verification |
Why hybrid covers more tasks
The streaming loop holds the "head" of the chain and serves fresh events, while the batch loop catches up on history and recalculates segments when parsing logic changes.
This separation removes the main contradiction: the live interface does not wait for the entire history to be reindexed, and the rebuild does not slow down the intake of new blocks.
The key element here is write idempotency and checkpoints by block height. If processing fails in the middle of a range, the loop resumes from the last recorded point rather than starting over.
This turns a failure from "reindex everything" into "catch up on what was missed."
The limitations of hybrid are organizational, not technical: discipline in the data schema and a single point of verification are needed, otherwise the loops begin to diverge.
We build this in at the design stage, so the mode is chosen for the task, not for ease of implementation.
How we build a custom blockchain indexer
We start with an infrastructure audit: we look at which nodes are already up, how synchronization with the network is configured, which blockchain requests are being made today and which of them are slowing down the product.
At this stage we fix the target metrics — indexing depth, allowable lag behind the latest block, the set of fields in the response — so that we can later verify the result by numbers, not by feel.
- Infrastructure audit. We examine the current nodes, storage, load and points of failure. We determine what to reuse and what will have to be replaced, and where the indexer will hit a ceiling as the number of requests grows.
- Pipeline design. We design the data flow: block ingestion, event parsing, enrichment, writing to storage, serving via API. We build in reprocessing on failure in advance — so that a missed block does not leave a gap in the data.
- Indexer setup. We bring up the services, configure event selection rules and storage schemas for the product's specific requests. Separately, we set up lag monitoring and alerts when synchronization stops.
- Completeness verification. We reconcile the indexed data with the network data on control segments: how many events were missed, whether balances and states match. We run load scenarios and check behavior on restart.
- Handover and support. We hand over configurations, schemas and documentation, and train the customer's team. After that we provide support: we monitor network updates and adjust the indexing rules.
Each stage ends with a verifiable result, so any deviation is visible immediately, not in production. Describe your task — we will come back with an estimate and a work plan.
Artifacts and documentation after implementation
After the indexer is launched, the team is not left with a "black box": everything that describes the system's behavior is handed over as working materials.
Using them, your engineer will be able to bring up a stand from scratch, replicate the configuration, and understand why the processing is designed the way it is.
- Indexer and node configurations: network connection parameters, reprocessing depth during reorganizations, batch size limits, retry settings.
- Data flow diagrams: from block ingestion to writing to storage and serving via API — with the points where delays and losses are possible.
- Maintenance regulations: scheduled restarts, reindexing when the schema changes, procedures for when the state diverges from the network.
- Access policies: roles for reading and writing, endpoint-level separation, rules for rotating keys and secrets.
- Storage schema description: tables, indexes, partitioning and stale data cleanup strategy.
- Set of checks: health endpoints, metrics for lag behind the network head, alerts for processing stoppage.
- Deployment instructions: order of bringing up components, dependencies, environment variables, resource requirements.
- Decision log: which alternatives were considered and why the current option was chosen — so that future changes don't break the logic.
The materials are handed over in your team's repositories and wiki, not in chat. We stay in touch during the stage when your engineers make their first edits — this reduces the risk that the documentation will diverge from the actual configuration.
Case study: how we cut indexing latency by a factor of three
The client — a network analytics service built on an EVM-compatible blockchain — came with a symptom rather than a task: the data in their dashboards lagged behind the network, and in the mornings the indexer would fall over during reprocessing.
They had written their own indexer for the job, but it read blocks strictly sequentially, on a single thread, and stored state in a single table with no partitioning. At peak load, the queue of unprocessed blocks grew faster than the worker could clear it.
What it was before
The delay between a block appearing on the network and its availability in the API reached 40 minutes at peaks.
Reorgs deeper than two blocks broke the state: the indexer did not roll back already-written events, and "dangling" records from the abandoned branch remained in the database.
Every few days the process would crash on memory — a restart meant a full recomputation from scratch and several more hours of downtime.
What we changed
We separated reading from writing: a dedicated collector pulls blocks in batches and puts them into a queue, and a pool of workers processes it in parallel by ranges.
We introduced explicit reorg handling — when hashes diverge, the indexer rolls back the affected ranges using the journal and re-reads the canonical branch.
We laid out the state across partitions with checkpoints: after a crash, the process resumes from the last confirmed height rather than from scratch.
What it became
| Metric | Before | After |
|---|---|---|
| Indexing latency at peak | up to 40 min | 12–13 min |
| Failures due to reorgs | regularly | class eliminated |
| Downtime after a failure | hours, full recomputation | seconds, from a checkpoint |
Latency was cut by a factor of three, and manual restarts disappeared from the process entirely. We also exposed the indexer's own metrics — queue depth, height lag, number of rollbacks — so the client's team could see degradation before users noticed it.
How does a custom indexer differ from off-the-shelf solutions?
Off-the-shelf indexers are universal pipelines: they read blocks, decode standard events, and lay them out into their own storage schema. As long as you work with typical contracts and a single network, that is enough.
The problem starts where your product's logic diverges from what an off-the-shelf tool can deliver "out of the box": non-standard contracts, custom data structures, recalculating state from historical blocks, combining multiple networks into a single query.
Custom development starts with the question "which queries must execute fast," not "which data is in the blocks." Engineers design the storage schema around specific scenarios: aggregates by address, balance history at an arbitrary height, links between events from different contracts.
An off-the-shelf solution will not offer such a schema — its model is fixed, and you adapt the product to it, not the other way around.
The selection criteria are simple. If you need a network overview, standard transfers, and events from popular contracts, an off-the-shelf tool will cover the task faster and with lower maintenance cost.
If you need custom metrics, non-standard aggregation, deep history, or multiple networks in a single response, a universal pipeline will hit its limits, and workarounds will consume more resources than building your own.
Chain reorganizations and node failures deserve separate mention. In a custom indexer, the logic for rollbacks and reprocessing blocks is designed around your data: it is clear which records are recalculated and which remain valid.
With off-the-shelf solutions this is a black box — when a discrepancy occurs, you have to figure out exactly what was lost.
The comparison should be based not on the number of supported networks, but on how quickly you can get an answer to a new question about the data.
An off-the-shelf tool answers questions that are already built in; a custom one answers the questions your product will have tomorrow.
Common technical concerns during implementation
An indexer is not a one-time integration, but a service that lives alongside the network: contracts change, events are added, node updates are released.
That is why the first thing a technical customer checks is not "will it work", but "what will happen in six months". Below are the concerns that come up on every project, and how we address them engineering-wise.
How difficult is it to maintain a custom indexer on your own?
The difficulty is not in the code, but in operations: reorgs, lagging blocks, database growth, reindexing after a contract upgrade.
We deliver the service with metrics (indexing lag, queue depth, error rate), alerts for lagging, and a reindexing procedure — the team can see the state without reading the source code. Updating the event schema is done via migration, not by rewriting from scratch.
Is it compatible with our current infrastructure and nodes?
The indexer connects to your nodes or to external providers through standard interfaces, and delivers the result to a familiar storage — a relational database, a search engine, or a queue.
On the input — regular RPC calls and event subscriptions, on the output — a schema that your backend consumes. Migration does not require stopping current services: the new index is populated in parallel, and switching is done by changing the read source.
How is security and data integrity ensured?
Data is taken from confirmed blocks with finality in mind, so events from reverted branches do not enter the index.
Access is segregated: the indexing service has only read access to the node and write access to its own schema, and signing keys are stored separately from the processing process.
Each event is accompanied by a reference to the block and transaction — any record can be verified on-chain.
What if the network or contract changes after launch?
A contract upgrade is a standard scenario: the new version of events is described by a separate schema, and old data remains accessible. When the format changes, we add a handler and run reindexing of the required block range without touching the rest of the history. Schema versioning is built in from the very beginning, so the transition happens without downtime.
Describe your network and the volume of events — we will get back to you with an estimate and an implementation plan.
Let's discuss your task and propose a work plan
To propose a work plan rather than generalities, we need to understand your task at the data level: which networks and contracts we index, which events and method calls matter, and what exactly should end up in API responses.
Describe the scenario — where requests come from, which queries are critical for speed, whether historical depth is needed or only fresh blocks. Based on that description, we'll draft an architecture and come back with an estimate of timelines and phases.
In the first conversation, we clarify requirements and boundaries:
- data sources: which networks, nodes, or providers, whether access to archive nodes is needed;
- data model: which entities, relationships, and aggregates should be in the index, how often they are recalculated;
- load and latency: how many requests per second are expected, what latency from block to response is acceptable;
- resilience: what we do during a chain reorganization, a missed block, a node failure — whether automatic catch-up is needed;
- access and infrastructure: where we deploy, your own servers or cloud, who owns the keys and data;
- integrations: which services and teams will read the index, in what format it is convenient for them to receive data;
- evolution: which metrics and queries will appear later, so we can build them into the schema without rework.
Next, we propose the following steps: we record the agreed requirements in a short document, show a data flow diagram and the composition of the index, and break the work into phases with a verifiable result at each one. You see what is done and in what order, where the first version can be launched, and how it scales.
Describe the task in free form — we'll come back with an estimate and a work plan.