Why parsing blockchain explorers becomes a bottleneck
When a transaction index is built on top of public blockchain explorers like Etherscan, everything runs into someone else's rules. Parsing blockchain explorers relies on request limits on external services: daily quotas, rate windows, mandatory cursor-based pagination.
Over long ranges, this produces a fragmented block history — traversal breaks midway, and a retry from the same cursor returns a different slice of data.
The second problem is incomplete transaction data. An external service may not return internal transfers, contract events, or finality status, and some addresses are marked as executable code only after it appears on the network.
As a result, the same block is described differently by two sources: the amounts match, but the set of operations does not. The team's time goes not into analytics but into manual reconciliations and investigating discrepancies.
Then the indexing bottleneck kicks in: while data is still loading, reports, notifications, and wallet balances show yesterday's state.
A single missed block propagates through the entire chain of dependent calculations and surfaces at the client — in an export, reconciliation, or risk decision.
We have been down this path: we started by assembling datasets through public interfaces and hit exactly the same ceilings. Let's break down how this is addressed and where the external service ends and the environment you control begins.
What a managed blockchain data processing pipeline provides
Data flows unevenly through public explorers: the collection pace is set not by our pipeline but by an external request limit. Once the threshold is exceeded, some requests are rejected, and if the handler does not keep a log of ranges, those blocks fall out of the registry.
The error is not exposed immediately but during balance reconciliation weeks later, when recovering the gap is more expensive than collecting the data again.
An in-house node removes this class of failures. We read blocks directly from the event log and keep a pointer to the last processed block, so the external provider's limits no longer set the pace.
Collection runs continuously: the queue is filled from the node, not from another service's responses. Etherscan and similar explorers remain useful for spot-checking transactions.
The integrity of the registry rests on idempotent writes: the record key is the transaction hash and event index, so reprocessing a range does not create duplicates.
The run log stores the boundaries of each range and its status, so any segment can be replayed selectively without restarting the entire pipeline.
The effect is measurable across three metrics: historical backfill time, share of lost records, and number of failures due to limits. The first drops severalfold due to parallel reading; the second goes to zero — a gap is visible in the log, not through a mismatch in totals. Address and balance reconciliation stops being a one-off audit and runs on a schedule.
Data collection methods: public interface, own node, archive
The choice of access method determines not only development speed but also which questions can be asked of the data at all. A provider's public API covers typical scenarios in a matter of days but makes you dependent on someone else's limits and method set.
An in-house node gives you control over requests and history but requires resources for synchronization and maintenance.
The key fork is whether historical depth is needed.
Almost any source serves fresh blocks and current balances, but state at an arbitrary height in the past is available only where block archives are stored: an ordinary node prunes old states, and some requests simply cannot be fulfilled.
That is why sources are often combined: a public API for operational tasks, an archive for analytics and reconciliation.
| Method | When it fits | Limitations |
|---|---|---|
| Provider public API | First product versions, wallet scenarios, tracking fresh transactions | Request limits, incomplete history, method set and depth change at the provider's discretion |
| Provider streaming subscriptions | Real-time event monitoring and reaction | The past is unavailable; a dropped connection must be caught up with a separate historical pass |
| In-house node | Independence from external limits, your own indexing rules | Long initial synchronization, growing storage volume, ongoing maintenance |
| Block archive on your own node | Analytics, audit, state queries at any height | Storage grows noticeably faster than an ordinary node, higher hardware requirements |
| Your own aggregation layer on top of the archive | Complex queries, reports, and metrics by your own rules | Requires team support, and discrepancies with third-party source data must be reconciled |
Provider limitations should be checked before launch: which methods are available, what the historical depth is, how the service behaves under peak load, and whether it warns about changes.
We record this in the source schema — so that migrating to your own node is a planned step, not an emergency one.
Implementation stages: from task audit to handover to support
The order of work is the same for a single network and for a multi-network environment: first we record exactly what needs to be read from the chain, then describe the flows, and only then deploy the infrastructure.
This eliminates the situation where the node is already up and synchronized but the data does not match the explorer.
- Data source audit. Which networks, contracts, and event types are needed, what the public API returns, where rate limits exist, and what the available historical depth is. The output is a list of sources with their limitations.
- Flow schema. We separate what is read directly from the node and what comes from the API, where intermediate storage sits, and how chain reorganizations are handled. We agree on the schema before development starts.
- Deployment and configuration. We bring up a full or archive node, configure indexing, queues, and retries on failures. Writes are made idempotent so that a repeated pass does not duplicate events.
- Verification by reconciliation. We compare the export against a reference — an explorer or a second independent source, check block-level checksums and final balances. Any discrepancy is investigated before launch.
- Load and fault tolerance. We run it at historical depth, check behavior when the node is unavailable and recovery after a break, and record catch-up time after downtime.
- Handover to operations. We provide procedures, a monitoring dashboard with lag alerts, on-call instructions, and team training. After that comes warranty support: we fine-tune edge cases and stay in touch.
Schemas, access policies, and regulations: what remains in the environment
The value of a parsing environment is not that it works today but that it can be reproduced and repaired by other hands. Therefore handover includes not only services but artifacts: configurations, schemas, and rules by which the system lives after launch.
- Source map — what data comes from where: public explorer interfaces, in-house nodes, backup providers, and what substitutes for what during a failure.
- Flow schemas — the path of an event from block to storage and marts: where normalization happens, where deduplication happens, where requests are retried, and where order can be lost.
- Node configurations — networks, storage depth, connection limits, indexing mode, and an explanation of what changes as load grows.
- Access policies — service roles, keys, network rules, read and write permissions, and separately the access revocation procedure.
- Failure response procedure — a "symptom → check → action" table: indexing lag, chain gap, provider unavailability, data discrepancy.
- Metrics and thresholds — which signals are considered normal, which are critical, who receives the alert, and where.
- Operations instructions — how to deploy the environment from scratch, how to reindex a segment of history, how to connect a new network.
With this set, the customer's team can add a network or change a provider according to the instructions without re-analyzing the architecture from scratch.
Case study: eliminated downtime and sped up data reconciliation severalfold
The customer — an analytics service for the Ethereum network — collected history through public explorer interfaces, i.e., ordinary parsing of blockchain explorers.
At peak loads, requests hit limits, and during chain reorganizations some blocks were lost, creating gaps and duplicates in the database.
The share of lost records reached 6–7%, and discrepancies with the network were found only manually — about 30 reconciliations per month, each on exports.
We built an in-house pipeline: our own node as the source, a stream of blocks and events, and writes keyed by "block height + event address" — so reprocessing a range does not create duplicates.
During chain reorganizations, the range is reread automatically, failed requests go into retry with increasing delay, and lag behind the chain head is visible on a graph and raises an alert before data starts to diverge.
We added automatic reconciliation: checksums by range are compared against an independent source, and discrepancies are broken down by type — miss, duplicate, incorrect field.
Manual reconciliations dropped to 1–2 per month instead of 30, and each takes minutes: an engineer reviews a discrepancy report rather than comparing raw exports.
A full historical pass was reduced from 14 hours to 40 minutes thanks to parallel range processing and local reads instead of external requests. An entire class of failures disappeared: losses due to public interface limits and "holes" after network reorganizations.
On the live stream, the share of lost records stays at zero, and the daily export matches the network without manual intervention.
How a public interface differs from your own node
A public interface is someone else's infrastructure: someone else holds a copy of the chain and serves you data on request. Your own node is the same copy of state, but yours, and you decide what to index and in what volume.
The difference shows up in three places: the depth of available history, stream behavior under load, and where the connection breaks during a failure.
In terms of depth, public gateways almost always serve only a recent window of blocks and events — deep queries are rejected or truncated by range.
An archive node on your side stores history from the first block, so recalculating metrics for past periods and restoring missed events does not hit someone else's limit. This is exactly what determines whether the index can be backfilled or data can only be collected forward.
Under load, a public interface imposes rate limits, responds with variable latency, and delivers different quality across gateway operators.
Your own node is limited by disk and network, not by someone else's limit: you set the request parallelism, batch size, and cache depth yourself and see what you are hitting.
A public gateway is an external point of failure: changed terms, reindexing on the other side, or node degradation stop collection without your involvement.
Your own node moves responsibility inward: you need chain head lag monitoring, reorganization handling, and idempotent writes so that a repeated pass does not double events.
The choice is usually not binary: a public interface covers reconnaissance and one-off tasks, while your own node covers regular collection with full history.
What is clarified before launch: limits, history, failures
Before building a parsing schema, engineers record the boundaries of available data: which external services provide history, at what request limit, how far back it extends, and how complete it is.
Without this inventory, the pipeline fills up until it hits the first limit or discovers a depth gap already at the analytics stage. Below are the doubts that most often come up at this step.
An external service changes limits or closes access — what then?
External service limits are not constant: they are lowered, paid tiers are introduced, and old methods are sometimes disabled. That is why access is encapsulated in a separate adapter layer — changing the source or limit does not affect processing logic. Plus, we always provide our own data source as a backup path.
Available historical depth will not be enough for our tasks?
Public interfaces usually serve not the entire chain but a window from several months to a couple of years. If earlier states are needed, we bring up our own node and read blocks sequentially, or fill in the missing segment from archival sources. The decision is made for a specific horizon, not "just in case."
How can we make sure the data is complete and without gaps?
Completeness checking is built on reconciliation: continuity of block numbers, matching checksums across a sample, and convergence of aggregates with an independent source.
Discrepancies are logged separately rather than dissolving into the general stream — so a gap is visible immediately, not a month later in a report.
Can we start small and expand?
Yes, and that is the right order. First one contract or one event type, debugging on it, then gradual expansion of coverage — new networks, tokens, historical periods.
Each step is added without rewriting the previous one, so storage volume grows predictably and stays under control.
Let's discuss the task: what we need from you for an estimate and plan
To come back with an estimate and plan, we need not a general idea but inputs on data and infrastructure. Below is a checklist: if some items are not yet defined, we will record assumptions and clarify them during design, not in the middle of work.
- Which networks and entities are needed: transactions, token transfers, internal calls, contract events, balances — and in what form to expose them externally.
- Where we get data from: public explorer APIs, providers with ready-made indexing, or an in-house node with a full archive.
- Volumes and update frequency: historical depth, stream of new blocks, acceptable lag, and recalculation window.
- Storage and query schema: where we store, who reads, by which keys queries are built, what volume is kept hot.
- Access and roles: API keys, node credentials, write permissions to storage, who on your side approves changes.
- Accuracy rules: how we reconcile gaps and discrepancies during chain reorganizations, what we consider the reference source.
- Delivery mode: one-time snapshot, continuous stream, or regular recalculations; whether replay and rollback are needed.
Describe the task in free form and attach what you already have — we will discuss solution options. After that we will come back with an estimate and work plan: stages, timelines, set of checks, and points where your confirmation is required.