Why the TON indexer cannot keep up with growing load
The TON indexer starts falling behind not at peak load, but earlier — when the event stream from the network is consistently faster than the confirmed write to storage.
At first the gap is measured in single blocks and is not visible in API responses, then it grows avalanche-like: the queue becomes longer than the window that workers can process in one pass.
There are four typical causes, and all of them concern the processing cycle design, not hardware capacity.
Block reprocessing: after a branch reorganization, the worker often replays the entire range rather than only the affected transactions. Every missed block in the queue is paid for with double work, and intermediate states in storage then have to be verified separately.
Competition for nodes adds its own effect: with a limited number of access points, some requests fail due to limits, different workers read state at different moments, and balance discrepancies surface in the API.
A separate issue is the key schema. A key like "address plus date" spreads records across hot partitions, batch writing does not come together, the writer hits disk operations — and lag grows at the output stage.
A shard-range key with batch accumulation gives a steady stream and predictable delay between an event in the network and its appearance in storage.
Missed blocks are dangerous not by themselves, but because catching up on them competes with the current stream for the same nodes and disk. That is why lag is measured as a separate metric for each shard, not as a service-wide average, and decisions are made based on it.
We break down how this is addressed: where to set the boundaries of the reprocessing window, how to separate reads and writes between nodes, and which key format keeps writes steady.
What managed indexing delivers: measurable effect
A managed TON indexer keeps lag behind the network head within one or two confirmed blocks — instead of minutes of waiting after each transaction. Reorganizations are handled by replaying the affected height range, not by fully reindexing the entire history.
History completeness is maintained by a journal of processed masterchain blocks: for any height, account state, its events, and transfers can be restored.
Correctness is checked by comparing state checksums at random heights — discrepancies are visible before the user notices them.
Resistance to reprocessing is provided by idempotent writes: a key made from block height and transaction hash does not allow data duplication. A worker that crashes midway through a range continues from the last checkpoint and loses no events.
Node load is reduced by subscribing to new blocks instead of constant polling, batch requests, and a hot account cache — node requests drop by an order of magnitude.
The freed headroom goes to deep historical queries: responses by address or token return in fractions of a second even across a long height range.
Indexing modes: comparison of approaches and limitations
The choice of mode determines not only how quickly data appears in the API, but also how the TON indexer survives reorganizations and gaps.
There are two input parameters: the acceptable delay between block finalization and its visibility in the response, and storage depth — how much history needs to be retrieved on request.
| Option | When it fits | Limitations |
|---|---|---|
| Streaming indexing | Front-end apps, wallets, balances, and transaction statuses: the response is needed almost immediately after finalization | A buffer for reorgs and idempotent writes by block key are required; a parser failure stops the entire stream, and history cannot be recalculated retroactively |
| Batch recalculations | Analytics, balance reconciliation, storage schema migration, recovery after failure | Data appears with a delay of tens of minutes; a repeat pass over the node archive is expensive in disk and storage load |
| Hybrid scheme | Public TON indexer API, where both fresh data and deep queries matter | More complex to operate: two pipelines, separate checkpoints, and coordination of the boundary between hot and cold layers |
Storage depth is a separate axis and should not be confused with the mode.
We keep fresh blocks in fast storage, and history in compressed storage, with a pre-defined retention policy: how many recent blocks we guarantee for rollback, what goes to the archive, and what is recalculated from it on demand.
The hybrid usually becomes the working option: streaming provides low latency, while batch passes close gaps and populate historical slices.
In practice, the mode is selected for the query profile, not the other way around: first we look at which queries prevail, then we set latency and depth.
Stages of TON indexer implementation
The workflow is the same for a new TON indexer and for replacing a custom script that can no longer keep up with the block stream. First we determine which data queries the product needs, and only then choose the storage schema: if you start with code, you have to rewrite midway. Below is the sequence with acceptance points at each stage.
- Data schema audit. We review tables, relationships, and heavy queries, noting missing composite indexes and duplicates. The output is a source map and a migration plan.
- Storage model design. We decide what to keep normalized and what in denormalized projections for frequent queries: address history, balances, contract events.
- Queue setup. We divide the block stream into ingestion, parsing, and writing; the queue absorbs peaks, and after a failure processing resumes from the last confirmed position.
- Consistency check. We compare the index with network data by balances, transaction counts, and reprocessing; we reduce discrepancies to zero on a test range.
- Load run and cutover. We monitor indexer lag on historical and real-time data, migrate traffic in stages, with rollback to the previous pipeline.
- Handover of runbooks. We hand over schema and query descriptions, restart and recovery instructions, and metrics with thresholds. Further operation is handled by your team.
Deliverables: schemas, runbooks, tools
After launch, the customer's team retains not only a working service but also a set of documents by which it can be operated and developed without us.
We hand over the data schema description, runbooks, and observability tools — so that the on-call engineer understands the cause of a failure, and a new developer can add an API method without parsing someone else's code.
- Data schema — which tables store blocks, transactions, and events, what key is used for partitioning, and which indexes cover frequent queries by address and time period.
- TON indexer API specification — list of methods, response formats, cursor pagination, limits, and error codes: it lets integration be built without clarifying questions.
- Access and roles — read-only service access, separate keys for analytics, administrative operations, rotation procedure, and isolation of staging from production.
- Restart runbook — sequence in which components are started, how the service catches up on missed blocks and from which point it resumes processing so events are not duplicated.
- Queue monitoring — metrics for queue depth, index lag behind the chain, and share of dropped messages, alert thresholds, and first on-call steps when triggered.
- Snapshots and recovery — how storage state is snapshotted, where it is stored, and how long it takes to return to operating mode.
- Troubleshooting typical failures — desynchronization, duplicate events, lagging node: what signs identify them and what to do step by step.
- Knowledge transfer — recording of an architecture review and answers to team questions so operations do not depend on one person.
Case study: how data completeness grew after a restart
What it was like before the restart
The indexer was built for quick launch: blocks were read in a single stream, everything was written to one table without sharding.
At peak, lag behind the network reached several hours, and during desynchronization the process silently skipped a range of blocks — gaps were found from complaints by services built on this data.
History completeness stayed around 88%: some internal transfers and contract events were not included in the index.
In TON this is critical — a skipped range breaks the chain of states for dependent addresses, and the real balance can no longer be distinguished from accumulated error.
What was changed and what became
Ingestion was rewritten to parallel shard processing with checkpoints, reindexing became managed: depth is set by a parameter — from recent hours to the entire network history. Failure recovery is now automatic, from the last confirmed point, without manual parsing.
- Lag behind the network at peak — minutes instead of hours.
- History completeness — 100% across verified ranges, checksums match for each shard.
- Failure recovery — about 20 minutes of automatic replay instead of hours of manual work.
- Reindexing depth — the entire history, not a window of the last day.
We did not compare by eye: we ran the index against a full node across several ranges, no discrepancies. After the restart, teams stopped maintaining their own exports — data from the index matches the network and is served immediately.
What to do if the indexer is already deployed and lagging
The TON indexer in operation lags for two reasons: the accounting schema changed — internal transfers were added, the field set changed — or volumes grew, and the block processing cycle no longer fits the interval.
The service cannot be stopped: balances and transaction history are needed constantly, and a run from scratch takes hours.
A workable technique is gradual migration through dual writes. The handler writes to both the old and new schema in one transaction, so states do not diverge. The read layer switches not entirely but by query groups: rollback amounts to removing a flag, not restoring from backup.
We migrate history in the background. We record the block height from which dual writes began — this is the divergence point. Everything earlier is loaded in batches into the new schema by parallel handlers with node load limits, so the live stream does not lag further. We catch up to zero divergence.
Schema change is safer than it seems: the new table lives alongside the old one, with no lock conflicts.
Verification is mandatory — a control sample of addresses and operations is run through both schemas, responses are compared automatically, and only after they match does reads move to the new one.
If lag has already accumulated, it is removed by rebuilding the hot layer from the block archive without touching accumulated data. Starting from scratch is not worth it: balance and trade history are lost, and recovery takes a day. We keep the old layer readable until the new one proves a match in verification.
Technical doubts before starting work
An indexer is a state that must match the chain under any failure: a missed block, connection loss, reorganization of recent blocks.
Below are answers to engineering questions most often raised during evaluation: integrity, reprocessing, access, and responsibility boundaries after handover.
How is data integrity confirmed for each block?
Each processed block is recorded with its hash, number, parent hash, and write timestamp. Verification proceeds from the last confirmed mark: if the parent hash does not match the one already saved, the batch of blocks is rolled back to the common point and processed again.
The final check is a checksum of state over a window of blocks; it is verified independently of the main parsing cycle.
What happens if the indexer stops for several hours?
Parsing follows a cursor, not time, so after startup processing continues from the last confirmed mark. The write operation is idempotent: reprocessing the same block does not create duplicates in storage. Lag is caught up in batches until reaching the current height, then normal mode resumes.
How are access rights to the deployed setup organized?
Access is granted by roles: read through the data interface, service operations for parsing and migrations — separately. Keys and credentials are transferred to your responsible person; we do not retain permanent access.
Revocation of access on our side is documented in writing and confirmed by a connection attempt from both sides.
How is responsibility divided after handover?
We hand over parsing configurations, storage schema, reprocessing policy, recovery instructions, and procedures for state divergence. Your engineers restore the pipeline using these materials without our help.
Warranty support covers parsing logic errors — they are reproduced and fixed; changes to your infrastructure and node access rules remain on your side.
Let's discuss your task and come back with a plan
So that project estimation is based on facts rather than assumptions, at the start it is important for us to understand the boundaries: whether history from the first blocks is needed or a window for recent periods is sufficient, which entities must be in the response, and what delay between a block appearing in the network and its visibility in the API you consider acceptable.
Two more parameters — where the data stream comes from (your own nodes or external sources) and what load headroom to allow so the indexer does not lag behind the chain head during peak bursts.
Based on the conversation, you will receive:
- a parsing scheme: shard traversal, processing order for the masterchain and workchains, behavior during reorganizations and block re-issuance;
- a storage model with index descriptions for specific queries — so that queries by address, transaction, and history maintain the required speed at your volumes;
- an API contract with response examples and fields, including pagination, filters, and finality indicators;
- a stage-by-stage work plan with acceptance points at each: raw stream, storage population, your own API, load tests;
- team composition with roles — who is responsible for parsing, who for the query layer, who for support;
- a handover-to-operation plan: dashboards for lag behind the chain head, reindexing runbook after failures, update procedure when block format changes;
- a list of risks and assumptions affecting timeline and scope.
Describe the task in free form — volumes, sources, requirements for responses. We will come back with an estimate, work plan, and team roles.