Blockchain parsing: where data collection from the network breaks
Collecting data from the network looks like a textbook task: connect to a node, fetch blocks, parse transactions. In practice, skipped blocks and balance mismatches surface weeks after launch, when the figures in reports have already diverged from the chain state. We break down where blockchain parsing breaks and why retrying the request does not fix it.
Skipped blocks are rarely related to the network itself. A node limits the range returned per request, the connection drops midway, and the restart begins not from the saved cursor but from the last block in memory.
If the checkpoint is written asynchronously, after a crash the processes diverge, and the gap in history is discovered only by reconciling totals.
Balance mismatches are caused not by a transfer but by what is not visible in the transaction. Internal transfers, contract calls, and events from logs need to be parsed separately — otherwise the balance reconciles for the native coin and diverges for tokens.
A separate pain: non-standard token implementations and different numbers of decimal places, when the amount exists but there is nothing to interpret it with.
Third area — chain reorg. Blocks already written to the database fall out of the canonical branch, and recalculations on top of them hang and drag the entire aggregation along.
This is closed by waiting for finality, idempotent writes, and deduplication by a key that includes the event index, not just the transaction hash. Below — how we build a pipeline where these failures are visible before the numbers diverge.
What verified collection and parsing of network data provides
We check the result against four measurable criteria — they are visible in monitoring, without manual log analysis. If at least one declines, parsing formally works, but the data for reporting and reconciliations is already unreliable.
- Data freshness. Lag from the chain head is measured in units of blocks, not tens, and does not grow under peak load: the interval between block appearance and writing to storage remains stable.
- History completeness. In the processed range there are no missed heights: every transaction, balance change, and contract call is tied to its block, and the discrepancy with the source by number of records is zero.
- Resilience to reorgs. When the chain rolls back several blocks, records from the canceled branch are marked invalid and recalculated — reporting does not "drift" after a reorg.
- Node load. Reading proceeds in batches by ranges, with cursor persistence and retries on interruption: the node is not overwhelmed by requests, and backfilling continues from the last processed height.
- Verifiability. Lag, number of gaps, and share of retries are exposed as metrics, so data quality is confirmed by numbers, not promises.
We test the parsing scheme on real networks — from Ethereum and compatible ones to networks with a different confirmation model.
We select the set of control reconciliations for each case: in some cases, comparison with a public block explorer is enough; in others, a periodic recalculation of a selected history segment in full is needed.
Blockchain parsing modes: streaming, batch, on demand
The data collection mode is determined by the task, not the other way around. Streaming block processing is needed where freshness matters: an event appears immediately after confirmation and goes into processing without waiting.
Batch history loading — when you need to restore state over a long segment or reindex data after a change in parsing logic. On-demand requests cover spot checks where a full pass does not pay off.
| Mode | When it fits | Limitations |
|---|---|---|
| Streaming block processing | live dashboards, alerts, address monitoring | requires resilience to chain reorgs and gaps, state is stored separately |
| Batch history loading | initial population, reindexing after a change in logic | long run, node limits on sampling depth, idempotency of retries is mandatory |
| On-demand request | rare spot checks, balance or contract call at a point in time | does not scale by volume, latency depends on node availability |
| Contract indexing | regular queries by events and addresses, analytics | requires a storage schema and migration rule, database volume grows with history |
How the modes combine
In practice they are combined: a batch pass populates the index by contracts and addresses, the stream catches it up with new blocks, and spot requests read the already assembled mart.
Common requirements for any scheme — idempotency of retries, a checkpoint of the last processed block, and separate handling of chain reorgs. Without this, balance mismatches and missed events surface on the reporting side.
How we implement a parser: stages of engineering work
Work proceeds in iterations of two to three weeks: each one closes one data source and ends with a verifiable artifact — not a report, but a working index that queries can already be built against. The order of stages is the same for a single network and for a set of several.
- Source audit. We inventory where data comes from: a full node or archive node, whether it serves historical states, whether contract event logs are available, whether throughput is sufficient for the required history depth. Here we also fix the list of target entities — transactions, balances, contract calls — and what is considered mandatory versus optional in them.
- Indexing schema design. We build the schema from queries, not from block structure: we partition tables by height ranges, create separate indexes for address, token, and internal transfers. We decide in advance what we store hot and what we aggregate.
- Collection and normalization. We set up handling of chain reorgs and reprocessing of affected ranges, make writes idempotent by block and operation key. We maintain a separate cursor for lag from the chain head.
- Completeness verification. We reconcile the result with an independent source: block checksums, number of events, final address balances. Every discrepancy goes into a review queue with the raw response saved, so the cause can be reproduced.
- Load runs. We run reindexing of a large historical segment and parallel queries, look at where it breaks, and configure retries and rate limiting for node requests.
- Handover of procedures. We provide maintenance instructions: how to add a new contract, what to do when the cursor lags, how to replay a range after a failure, and how to roll back the schema.
The key to predictability is that each stage has a measurable acceptance criterion: depth of indexed history, share of reconciled blocks, acceptable lag. They also later show that after changes in the network, the parser continues to reconcile.
What we hand over: configurations, schemas, procedures
We hand over not a "black box" but a working environment that your engineer can run, read, and modify without us.
For this, the delivery includes a fixed set of artifacts — from configurations to procedures describing not only normal operation but also recovery after failure.
- Working service configurations: node connection parameters, block confirmation depth, batch size for recalculation, limits of parallel tasks.
- Block traversal scheme: starting height, checkpoint storage, behavior on range gaps and on chain rollback.
- Data flow scheme: path from source to normalized tables, places for deduplication, enrichment, and serving via API.
- Storage model: description of tables and relationships, indexes for queries by address, contract, and period, archival policy for old snapshots.
- Access policies: roles and separation of read and recalculation rights, key rotation procedure, call log.
- Parser documentation: field purpose, event format, decoding of internal statuses, examples of typical queries.
- Recalculation procedure: when to run a repeat pass, how to limit load so as not to disrupt current synchronization.
- Set of checks and metrics: lag from the network, share of parsing errors, balance mismatch, plus an incident instruction.
Acceptance is practical: we deploy the environment on your side, run a historical segment, and reconcile checksums across a sample of addresses. If the discrepancy exceeds tolerance, we investigate the cause before handover, not after.
Case: indexing a network without skipped blocks
What it was before
The client — an analytics service for a loaded network: address balances, transaction history, contract statistics. Blockchain parsing was built on polling public nodes "for the current height" — whatever was received was written.
When a node lagged or a reorganization occurred, blocks silently fell out: the height gap was found from user complaints, not from monitoring.
In a sample reconciliation of 500 addresses, discrepancies with the independent explorer Etherscan occurred in 3-4% of cases, and on peak days — up to 9%.
What was changed
The pipeline was rebuilt: instead of polling "for the latest height" — subscribing to a block stream plus background catch-up by height ranges.
Each written block stores height, hash, and parent hash, so a chain gap is visible immediately and is not masked by the next block. Reorganizations are handled by rolling back to the depth set by the network policy, with reindexing of the affected range.
The last confirmed height is fixed in storage — restarting the service produces neither gaps nor duplicates.
What became
Discrepancies on the same sample — zero. Completeness was confirmed by a continuous pass over all heights and page-by-page checksum reconciliation: the number of indexed blocks matched the independent source, and there are no "holes" in history.
The latency for a transaction to appear in the external interface dropped from minutes to seconds, and balance desynchronization left the class of support requests.
How an archive node differs from a light client
The difference is that a light client stores only block headers and a small set of confirmed states, while it requests transaction bodies and historical states from other nodes as needed.
An archive node holds the entire chain from the first block and can serve state at any height — this is exactly what is called history depth.
For blockchain parsing, this is critical. A light client will not reproduce a contract state at last year's height: intermediate states are not stored in it, and event logs are available only for the interval the node has managed to process.
Full recalculation of balances, restoration of token holders at an arbitrarily chosen block, or tracing internal calls simply cannot be performed on a light client.
The cost of an archive is disk and time. The volume of history grows linearly and never decreases, and initial synchronization from scratch takes days, not hours.
A light client starts quickly and takes almost no space, but pays for this with dependence on data providers and reduced depth — it is suitable for observing current events, not for historical queries.
Therefore, we always choose the data source based on the task, not convenience of launch.
If you need an audit of fund flows, reconciliation of snapshots at a specific height, or retrospective incident analysis — we raise an archive node, and on top of it our own indexer with a replica.
When it is enough to monitor new blocks and serve data in real time, a light client with an external provider as a backup channel is sufficient.
A hybrid scheme costs more in maintenance, but removes the main risk — silent loss of history, because of which recalculation has to be started from scratch.
Common technical doubts from customers
Customers usually have the same doubts: what will happen if the connection to the node drops, whether we will hit source limits, and whether addresses and amounts from parsed transactions will leak. Below — how these areas are arranged in our parsing scheme.
What happens if the node goes down or the network rolls back blocks?
The parser stores a checkpoint by number and hash of the last processed block, and each record is idempotent by the key "network + transaction hash + event index" — reprocessing a range does not create duplicates.
If the rollback goes deeper than the confirmation depth, the range is reread from scratch, and the retry queue picks up what was missed without manual intervention.
Will we create load on data sources?
We work through a pool of nodes with rotation and request rate limiting, and take log samples by block ranges rather than one transaction at a time. As the stream grows, we add our own nodes — public ones remain as a reserve in case of degradation.
How is data privacy ensured?
Parsing runs on our or your infrastructure, access keys are stored in a secret vault and do not end up in configurations.
Addresses, amounts, and links between wallets are not shared with third parties, exports remain in your environment, and logs are written without personal data.
How difficult is it to add a new network?
The core is separated from network adapters: storage, normalization, and data serving are reused; only the block retrieval and log parsing layer is added. For networks with a similar event model, this is an edit to one adapter; for non-standard ones — a separate parser.
Let's discuss your network data collection task
To come back with a timeline estimate and work plan, we need to precisely understand the data composition and task boundaries. We will close some of the inputs during the conversation ourselves, but the more specific the input, the fewer revisions at the acceptance stage.
- Purpose of parsing. What should be produced as output: a reporting mart, address monitoring, contract analytics, or export to your system. This determines the depth of blockchain parsing and the storage schema.
- Data composition. Which networks and entities are needed — transactions, token transfers, balances, contract events; for what period and how often to update.
- Access sources. Your own full nodes or external ones, request rate limits, whether there are keys to network explorers.
- Current environment. Where to put the result: existing database, storage, file exchange; what limits on load and access rights.
- Contract complexity. Whether standard events are enough or parsing of internal calls, non-standard tokens, and restoration of amounts from logs is needed.
- Reconciliation requirements. How critical completeness is: whether lag from the network is acceptable, whether recalculations are needed during chain reorganizations.
- Constraints. Industry requirements, separation of access to addresses, rules for storing links.
Describe the task in free form — we will discuss the details and come back with a timeline estimate, data composition, and a stage-by-stage work plan. If some answers are not ready yet, we will go through them in the first conversation and record them in writing.