Why subgraph development slows down as data grows
At first glance, subgraph development is a declarative schema: entities, event handlers, field mappings. A handler catches a contract event, writes an entity, and the client reads a ready-made result set instead of traversing logs via direct queries to smart contract data.
While there are dozens of events per day, the schema looks ideal — the bottleneck is hidden in the data layer.
It manifests when the frequency of events on the network grows: there are more transactions per address, each block carries a batch of logs, and handlers stop keeping up with chain heads.
Indexing lag accumulates nonlinearly: the queue grows faster than write throughput, and any change to the schema or mappings requires a reindex from scratch — hours and days of downtime on historical contracts. Blockchain data indexing stops being a background process and becomes a critical path.
The cost is a distorted picture on the dashboard. The interface shows a balance based on an incomplete set of events, a report is built from a snapshot that lags thousands of blocks behind, and reconciliation with chain state reveals discrepancies.
Worse, the discrepancy is silent: the data is valid in form, simply incomplete, and it is discovered during incident investigation, not on a chart.
Next, we break down how this is addressed: by splitting the flow into hot and archival entities, filters by contract addresses, batch writes, and checkpoints that show actual lag, not just the status of the latest block.
What a managed indexing layer delivers: the effect in numbers
Subgraph development in The Graph delivers a measurable effect not by itself, but through a managed indexing layer: the schema, event mappings, and reindexing rules are described in the manifest and versioned together with the contracts. Data is collected in advance, so an application query no longer waits for a network traversal.
Data assembly speed no longer depends on the volume of history: a full reindex completes in minutes, and new events are processed continuously as they arrive.
Response freshness is measured by the delay between an event on the network and its write to the index and stays within seconds.
Recovery time after a failure is reduced: the index comes up from saved checkpoints rather than being rebuilt from scratch, so manual reconciliation is not needed.
A whole class of failures goes away — desynchronization of application data with contract state, missed events during chain reorganization, duplicates after reprocessing.
Query load is distributed: repeated queries are served by cache, heavy aggregates are calculated once during indexing rather than on every call.
The share of stale responses drops to zero because the application reads a consistent state as of a specific moment, not the intermediate result of a traversal.
We expose metrics — assembly speed, response latency, share of recoveries without manual intervention — to monitoring so degradation is visible before user complaints. We fix threshold values in the operations policy, not in chat.
Approaches to subgraph development: comparison of options
Comparison of approaches to subgraph development comes down to two things: how non-standard the contract logic is and who is responsible for the index after launch.
All three implementation options produce a working index on The Graph, but they handle contract upgrades, event format changes, and growing query volume from external data consumers differently.
A standard schema covers common entities — tokens, transfers, balances — and can be deployed quickly if the project has no custom logic.
In-house assembly by the team gives full control over queries, but it depends on a single engineer who remembers the event handlers and their order. A custom schema is designed for a specific data model and remains workable when contracts change.
| Option | When it fits | Limitations |
|---|---|---|
| In-house assembly by the team | Contract logic is simple, there is an engineer who knows the event schema | Depends on one person, delays when the contract changes, no query review |
| Standard template-based schema | Standard tokens, transfers, balances, need a quick start | Does not cover custom entities, diverges from the real model during refinement |
| Custom schema with maintenance | Non-standard logic, multiple contracts, external consumers | Longer design, requires access to repositories and event descriptions |
Moving between options does not require stopping the service: the new version is indexed in parallel with the old one, and consumers are migrated to it after reconciliation of sums and counts.
Therefore, the approach is chosen not “forever,” but for the current stage — comparing approaches captures exactly what changes during the transition.
How we manage implementation: stages of engineering work
The implementation stages are structured so that each ends with a verifiable result, not a report of hours spent. The order is the same for a new subgraph and for migrating an already running indexing setup to our support — only the depth of analysis at the first step changes.
-
Contract and access analysis. We collect addresses, interfaces, and the list of events, and reconcile them with the actual network state.
Separately, we record who owns the deployment keys and whether there is access to historical data: without a block archive, indexing from scratch gets stuck at the very first stage.
-
Data schema analysis. We define entities, fields, and relationships from real queries — what the interface requests, what analytics needs. Unused fields bloat storage and slow responses, so we agree on the schema before writing handlers.
-
Handler design. Each event is turned into entity changes, accounting for block order, reorganizations, and repeated calls. We build in on-demand record creation upfront so data is not lost on first access.
-
Indexing setup. We set the network, start block, pruning of unnecessary ranges, and batch writes. We run a full sync on a test environment and measure the time to the current block.
-
Query correctness verification. We reconcile responses for control addresses with data directly from the network: balances, volumes, event counts. We resolve discrepancies before deployment, not after a user complaint.
-
Deployment and versioning. We publish with an explicit version — the previous one remains available for rollback and for clients that have not yet updated.
-
Project handover. We hand over the schema, configuration, update and access instructions: the team maintains indexing without our involvement.
Handover artifacts: schemas, policies, documentation
We hand over not only a working index, but everything needed to maintain it independently. The check is simple: your engineer makes a change to event processing and releases a new version without asking us questions. The package includes:
- Schema documentation: entities, fields, and relationships, mapping to contract events, indexes — with notes on what will need to be changed when a new event is added.
- Entity and field descriptions: attribute purpose, type, source in contract logs, behavior with missing values.
- Event processing rules: what is indexed, how duplicates, chain reorganizations, and contract upgrades are handled.
- Access policies: roles in the admin panel, separate keys for manual deployment and for builds, rotation procedure.
- Maintenance policy: version release procedure, reconciliation of the index with contract state, response to discrepancies.
- Team instructions: local launch of the node and indexer, common queries, reproducing a failure on a block range.
- Update and rollback procedure: how to pin the previous version so queries do not fail while the new one catches up on history.
Each item is validated at acceptance: deployment and version release are reproduced according to the instructions without the author's involvement. Maintenance remains with your team, and we get involved only for new tasks.
Case study: eliminated indexing lag and a class of failures
The starting point was a working subgraph on The Graph: the main contract and a factory as event sources. Queries were served, but analytics dashboards lagged the network by 6–9 hours, and every chain reorganization triggered a full reindex.
Where the bottleneck was
The handlers read contract state at the moment of processing instead of taking it from the event itself: each block required several network calls, and processing could not keep up with the chain head.
The second limitation was writing — entities were saved one per event, so storage held back the speed.
A separate class of failures was the lack of finality control. Events from unconfirmed blocks made it into the database, and after reorganizations discrepancies had to be investigated manually by reconciling aggregates with contract state.
What we changed and what measurements confirmed
The mappings were rewritten: we take data from the event, and external calls were moved out of the hot path into batch processing. Entities are collected in batches and saved in one pass, immutable fields were moved into a separate type, and the schema was cleaned of duplicates.
A confirmation threshold was added before writing, and during a reorganization processing restarts from the fork height.
After the migration, indexing lag is measured in seconds instead of hours, a full reindex takes three hours instead of forty-plus, and storage volume has been reduced by roughly three times.
The analytics dashboard responds an order of magnitude faster, and over a quarter of observation there has not been a single desynchronization after reorganizations; previously these were weekly incidents.
What to do if your own data collector is already deployed?
You do not have to shut down your own collector to move to The Graph: the new environment comes up alongside it, and consumers are switched over one by one.
What matters is not a comparison of tools, but the number of data readers and their freshness requirements — this determines whether two environments need to run in parallel or a full replacement will suffice.
The main difficulty is data migration. History cannot be transferred by exporting tables: every record must be reproduced from contract events, otherwise aggregates will diverge from the chain.
We reconcile the environments by block ranges — event count per contract, token sums, latest address state: a discrepancy in a specific range shows where indexing lost a block.
| Option | When it fits | Limitations |
|---|---|---|
| Full replacement | One consumer, downtime is acceptable | Reindexing history takes hours, a cutoff point by block number is needed |
| Gradual migration | Multiple consumers, different read schemas | Long period of dual writing and regular discrepancy reconciliation |
| Parallel operation | A critical environment cannot be stopped | Double load on nodes, record deduplication is needed |
Compatibility between environments rests on identical identifiers: transaction hash and log index, idempotent writes, a unified format for addresses and amounts.
Then a repeated pass does not create duplicates, and a consumer can be returned to its own collector without losing history.
We choose the path based on the criticality of the environment: where downtime is acceptable, we go with full replacement; where it is not, we use parallel operation with gradual consumer migration.
Data freshness, access, and limitations: common concerns
Concerns at the contractor selection stage are usually the same: how quickly the index catches up with new blocks, who exactly will get access to queries, what happens under a sharp load increase, and how painful it is to change the schema after launch. Below are short answers on each point, without marketing or generalities.
How fresh is the data a subgraph returns?
The indexer processes blocks as they are confirmed and relies on a configurable finality depth — it defines the trade-off between response speed and resilience to chain reorganizations.
We monitor lag by reconciling with an independent node: if the discrepancy exceeds a set threshold, an alert fires, rather than numbers in the interface silently going stale.
How is access to data separated?
We grant access using keys: a separate key per team or service, its own rate and query size limits, and an access log. A key can be revoked without rebuilding the schema, so removing a contractor or internal consumer does not require stopping the service and redeploying.
What happens when the number of queries grows?
Limitations are removed architecturally: pagination instead of full exports, precomputed aggregates in the schema instead of heavy history traversals, and caching of repeated queries. When one indexer is no longer enough, a second is added — consumers do not notice.
How is the schema updated without downtime?
A new subgraph version is deployed in parallel with the old one. Consumers migrate gradually, the old version remains available until the last client, and rollback is switching the pointer to the previous build. This way, a data model change does not turn into an outage.
If any of these concerns is not resolved, we will break it down on your schema: we will show how indexing, access, and versioning will map to specific queries.Let's discuss the task: what we will clarify and how we will come back with a plan
To make the project estimate substantive rather than guesswork, we need a set of inputs — they show the volume of mappings, the number of entities, and the indexing depth.
Below is the minimum list; if something is not available yet, we will advise where to get it and what can be substituted.
- Contract addresses and network: where the subgraph is deployed and which other environments need support — testnet or mainnet.
- ABI or contract sources — we use them to reconcile event signatures and cut off discrepancies between documentation and code.
- List of events to index and the rule for which of them are considered significant: this sets the volume of processed records.
- Entity and relationship schema: what questions are asked of the data and how they are aggregated — storage structure and query count depend on it.
- Typical queries coming from the interface or dashboard: filters, sorting, pagination, expected response time.
- Start block and deployment addresses — to avoid pulling service history into the index and to keep builds reproducible.
- Access to the node and the repository with the build: needed to run indexing on real data, not on a sample.
Next, we come back with a project estimate and work plan: stages, team composition, what we do and what remains on your side, and where the approval points are.
If the inputs reveal forks — for example, a public hosted subgraph or a private one on your own node — we will show both options and their limitations in terms of load and access. The project starts after the plan and acceptance criteria are agreed.