Development of Blockchain Event Indexing System

With typical queries like "show all user transactions for the last 30 days" or "DEX trading volume", eth_getLogs is slow. On an archive node from block 0, the query takes minutes; on a public RPC, it times out or returns block range too large. We design fault-tolerant event indexing pipelines with exactly-once semantics and real-time monitoring. Resource savings: up to 80% time on search queries, 10x reduction in RPC costs. Real case: a client saved tens of thousands of dollars annually on infrastructure. Our team has 8+ years of blockchain development experience and over 50 implemented projects. Get a consultation — we'll evaluate your project in 1 day.

Approach to Event Retrieval

There are three ways to retrieve events from a node, each with trade-offs:

Polling is the most common choice: a worker periodically queries the node for a range of blocks, storing lastIndexedBlock. On restart, it continues from the last processed block. WebSocket offers low latency but requires automatic reconnection and synchronization of missed blocks. Firehose is an enterprise solution for high-throughput systems.

Handling Chain Reorganizations

Reorgs are the main source of bugs. On Ethereum PoS, finality occurs after two epochs (~12 minutes). On BSC or Polygon, reorgs of 3-5 blocks are common.

Strategy: index with a delay of N confirmed blocks (13 for Ethereum), store the hash of each indexed block. If a hash mismatch is detected, roll back to the last matching state and re-index.

-- Indexer state table CREATE TABLE indexer_blocks ( block_number BIGINT PRIMARY KEY, block_hash VARCHAR(66) NOT NULL, indexed_at TIMESTAMPTZ DEFAULT NOW() ); -- Event linked to block for rollback CREATE TABLE indexed_events ( id BIGSERIAL PRIMARY KEY, block_number BIGINT NOT NULL REFERENCES indexer_blocks(block_number), log_index INT NOT NULL, tx_hash VARCHAR(66) NOT NULL, contract_addr VARCHAR(42) NOT NULL, event_name VARCHAR(100) NOT NULL, decoded_data JSONB NOT NULL, UNIQUE(tx_hash, log_index) ); 

When a reorg is detected, delete records with block_number >= reorg_depth and re-index from that point.

Event Decoding Nuances

ABI decoding is trivial with viem or ethers.js, but there are pitfalls. According to the Solidity documentation on events, complexities arise with:

• Indexed vs non-indexed: indexed parameters go into topics, non-indexed into data. An event with 3 indexed parameters takes 4 topics. Decoding topics for structs is impossible — data is hashed.
• Anonymous events: events without a topic — rare, require a non-standard approach.
• Proxy contracts (EIP-1967): events are emitted from the proxy address, but the ABI is from the implementation. You need to resolve the implementation via storage slot 0x360894a13ba1a3210667c828492db98dca3e2076cc3735a920a3ca505d382bbc. More details in EIP-1967.

Example decoding in TypeScript:

import { decodeEventLog } from 'viem' function parseSwapEvent(log: Log, abi: Abi): SwapEvent { const decoded = decodeEventLog({ abi, eventName: 'Swap', data: log.data, topics: log.topics, }) return { blockNumber: log.blockNumber, txHash: log.transactionHash, sender: decoded.args.sender, recipient: decoded.args.recipient, amount0: decoded.args.amount0, amount1: decoded.args.amount1, sqrtPriceX96: decoded.args.sqrtPriceX96, liquidity: decoded.args.liquidity, tick: decoded.args.tick, } } 

Database Choice for Event Storage

We use PostgreSQL with partitioning by block_number or TimescaleDB for high-frequency contracts. Example partitioning:

CREATE TABLE swap_events ( block_number BIGINT NOT NULL, event_timestamp TIMESTAMPTZ NOT NULL, event_data JSONB ) PARTITION BY RANGE (block_number); CREATE TABLE swap_events_0_5m PARTITION OF swap_events FOR VALUES FROM (0) TO (5000000); -- similar for next ranges 

TimescaleDB provides continuous aggregates for metrics: hourly trading volume, transaction count — no background tasks needed.

Monitoring and Alerts

Include metrics: indexer_lag_blocks, events_per_second, reorg_count. Alert when lag exceeds 50 blocks. Health is checked via a health endpoint (503 on threshold breach). Stack: Go/Rust for worker, PostgreSQL/TimescaleDB, Redis for state, Grafana + Prometheus.

Our Process: How We Work

1. Analysis: Study smart contracts, identify all events, determine emission frequency. Average time: 1-2 days.
2. Design: Choose approach (polling for simplicity, firehose for high-load), design DB schema and API. Create data flow diagram.
3. Implementation: Write worker in Go/Rust, set up decoding, reorg handling, partitioning. Typical timeline: 5 to 15 days.
4. Testing: Simulate lag, reorgs, high load. Use testnets (Sepolia, Holesky).
5. Deployment: Docker containers with orchestration (Kubernetes or Docker Compose). Initial indexing of the last 1000 blocks.
6. Monitoring and Support: Grafana dashboards, alerts, monthly reports. Within the first month — fast fixes.

Indicative timelines: basic indexing of one contract — from 5 days; complex pipeline with partitioning and monitoring — from 2 weeks. Cost is calculated individually — contact us for an estimate.

What's Included

• Architecture design: approach selection, DB schema, API specification.
• Pipeline implementation: worker, decoding, reorg handling, partitioning.
• GraphQL API based on subscriptions (Hasura or custom resolver).
• Monitoring and alerts: dashboards, lag metrics.
• Documentation: data diagram, scaling points, runbook.
• Client team training.
• 1 month of support after launch.

Order a turnkey indexing system development. Get a consultation — we'll evaluate your project in 1 day and propose the optimal solution. Contact us.