New token parsing: detection, analysis, and risks

Token parsing: where the data flow breaks

Subscribing to node events is the most fragile element of token parsing. The connection drops when a node restarts, a provider changes, or the network degrades, and the code does not find out immediately: the channel is closed, but the handler keeps waiting for messages. A silent subscription break is scarier than an explicit error — new token detection simply stops.

The cost is missing events at the start of a token's life. The pool is created, trading is underway, but the contract is not in the registry: history is assembled retroactively and is already incomplete. The first minutes provide the metrics on which decisions are later made.

The second failure point is registry desynchronization on restart.

The cursor of the last processed block and the token record are not saved in a single operation, and a crash between them produces either duplicates when the range is reprocessed or missing events if the cursor moves ahead.

With multiple sources, discrepancies are added: one returns an event twice after a retry, another in a different order, a third loses blocks under load. Without an idempotent key and reconciliation by checkpoints, the registry diverges from itself.

All three failures are treated the same way — with explicit state and checks, not hope for a stable connection.

Below we break down how this is closed: resubscription with backfilling of the missed range, an atomic cursor, and registry reconciliation against an independent source.

What managed collection changes: speed, completeness, resilience

Token parsing is not "listen to new blocks and write everything indiscriminately." The difference between a raw stream and managed collection is visible in four measurable values: latency to write, miss rate, convergence after failure, and volume of manual review. Below is exactly what changes.

Detection speed: seconds from block to record

The queue is processed in parallel with deduplication by contract address and creation hash, so a token enters the registry within seconds after block confirmation.

Previously, this window was measured in tens of minutes — both the first trades and the initial liquidity remained outside the database.

Registry completeness: misses are counted, not assumed

Every contract creation is verified by re-reading from an independent source, such as Dune, and any discrepancy goes into a report rather than into silence. The share of lost events falls within statistical error, and the list of misses is always reproducible.

Failure resilience: the registry converges after a restart

Checkpoints by block number and reprocessing of the disputed range provide convergence without manual repair.

After a crash, the collector reads the missed segment and reconciles the result with an independent source: the discrepancy is visible immediately, not at a quarterly reconciliation.

Less manual review: filters instead of line-by-line inspection

Automated checks cut off obvious junk and typical schemes — a contract without liquidity, closed owner rights, no ability to sell. The operator is left with a few percent of addresses instead of the entire stream, and attention goes to genuinely disputed cases.

Comparison of token parsing approaches: modes and constraints

The data collection mode sets two parameters for the entire system: the delay between a token appearing on the network and it entering your database, and the depth of history used to calculate metrics.

Hence the constraints — the option that wins on speed almost always loses on completeness, and vice versa.

In practice, one primary mode and one auxiliary mode are used: event subscription catches new tokens, while scheduled polling or an archive node closes gaps and recalculates history retroactively.

They must be mixed deliberately — otherwise duplicate records and discrepancies in numbers between reports appear.

Implementation stages: from source audit to handover

The work order is structured so that token collection does not turn into a set of scripts that no one can maintain. First, we establish where data comes from and what counts as a correct event, then we assemble the stream, and only then do we enable detection. Each step has an artifact that can be checked and handed off to another team.

1. Source audit. We inventory nodes, block explorers, event queues, and external lists: what returns data, with what latency, which fields are mandatory, and where duplicates occur. The output is a list of sources with a priority and a fallback for each.
2. Data flow diagram. We describe the path of an event from block to storage and notification: normalization, deduplication, buffering, and reprocessing points. The diagram immediately shows where the stream can lose a token or duplicate it.
3. Collection setup. We implement subscription and backfilling of the missed block range, rules for matching addresses to new pairs, and filters by minimum liquidity and contract age. Logs store the block identifier so any record can be reproduced.
4. Risk scoring logic. We add contract checks: minting rights, transfer freezing, creator's holding share, and distribution of the first holders. Each trigger gets a weight rather than a binary flag — this filters out false alarms.
5. Failure testing. We run scenarios: node unavailability, data discrepancies between sources, event spikes, and service restart. We check whether the stream recovers without manual intervention and whether blocks are lost.
6. Acceptance with handover of procedures. We verify completeness and processing latency against a reference sample, prepare operating instructions, incident response, and the procedure for adding new sources. The client's team then runs the system independently.

Configurations, data schemas, and procedures: what you get

We hand over not only a working service but also a set of artifacts that your team can use to operate and extend it without our involvement. Everything sits next to the code in the repository: configurations, schema descriptions, rules, and procedures. Below is exactly what remains in your hands after handover.

• Configuration files — network node addresses, the list of monitored sources, request limits, polling intervals. They are edited without rebuilding the image or restarting processing.
• Data schema description — tables for tokens, pools, transactions, and events, relationships between them, and indexes. We separately document why this partitioning key was chosen and how it affects query speed.
• Filtering rules — liquidity thresholds, minimum trading volume, pool age, removal of duplicates and honeypot contracts. Each rule has a comment: why it exists and what will change if it is disabled.
• Metrics dashboard — new token detection latency, false positive rate, lag behind the chain head, and queue length of unprocessed records.
• Operations documentation — how to bring up the service from scratch, how to reindex history for an arbitrary period, and how to connect a new source without stopping processing.
• Failure procedure — step-by-step actions for lagging behind the network, source unavailability, and data discrepancies with the reference node, indicating the responsible person and response time.
• A set of tests and recorded events — filtering rules are run against them after any changes, so that changes do not open a stream of junk into the database.

Before handover, we go through each item together with your engineers and answer configuration questions. Describe the task — we will come back with an estimate and a plan.

Case study: how we eliminated gaps in the event stream

The client is a new token analytics service that serves a card immediately after contract launch.

The original scheme was polling the event log of a public node with a filter by factory address: once per slot, a range of blocks was fetched and the found events were written to the database.

When the connection dropped, the window was simply lost, and during a chain reorganization, some already saved records rolled back — leaving "dangling" tokens in the database.

On a stream of about 200,000 events per day, the miss rate reached 3.2%, a record appeared after 40–90 seconds, and the on-duty engineer manually reconciled a sample 6–8 times a week.

What we changed

We replaced polling with a subscription to dedicated Alchemy nodes, and on top of it we put our own queue with strict ordering by block height.

After block confirmation, we write a checkpoint — height and hash; on a break, the subscriber backfills the missed range and deduplicates records by the key "transaction + event index in the log." Records are idempotent, so repeated backfilling does not create duplicates, and lagging chain branches are cut off before being written to the database.

The class of failures that was eliminated is event loss on connection drop and duplicates during chain reorganization: both situations are now covered by a checkpoint and deduplication rather than manual review.

Subscriber lag behind the chain head is exposed as a separate alert, so degradation is visible before a gap appears in the cards.

How does token detection differ from token analysis?

Token detection is recording a fact: a record of contract or pool creation has appeared on the chain. The detector needs a minimum — address, time of appearance, source transaction, and who sent it.

The task is binary: do not miss the event and emit a signal as early as possible, while the token has neither history nor holders.

The data sources here are a stream of new records from several nodes at once, not polling a single one: a single node lags and rolls back during chain reorganization, and the token falls out of the sample.

The detector's polling frequency is maximal — it runs continuously in a narrow window of the latest records and accumulates only the raw event log.

Token analysis answers a different question — what kind of record this is by its properties. It requires the issuer's rights to additional minting, sell availability, holder distribution, liquidity, and signs of honeypot contracts.

So the fact of appearance is not enough: calls to the contract's current state and retrieval of historical data are required — infrequent but heavy.

Hence the difference in polling frequency and storage. The detector polls continuously and keeps a short window; analysis is triggered and then updated on a schedule: holder and liquidity metrics do not change instantly.

The detector needs an event stream; analysis needs versioned state snapshots to see how the assessment changed over time.

Mixing these layers is a typical mistake: the detector becomes overburdened with checks and stops keeping up, while analysis gets a raw stream instead of a prepared event.

The boundary is fixed in the architecture: the detector publishes a minimal event, and analysis subscribes to it and enriches it from its own sources.

Technical concerns: constraints, risks, support

Token parsing relies on external sources, and none of them guarantees a complete picture: some signals will be noise, and some will arrive with a delay. Below are the questions that most often arise at acceptance, with honest answers about the constraints.

What happens to data when a node restarts?

The read position is saved after each processed batch of events, so collection continues from the last confirmed point. Duplicates are cut off by key, and the confirmation window is set separately — resilience to chain reorganization depends on it.

Why multiple sources if one is stable?

A single node lags over time or returns incomplete history — this is a property of the network, not a failure. We read from several independent sources and reconcile: a discrepancy is flagged for review rather than averaged out.

How far behind real time are the data?

The delay consists of confirmation time and queue processing — in practice it is seconds, and it is exposed as a separate metric. For urgent scenarios, an early signal is enabled with a "preliminary" label.

What about support after handover?

We hand over configuration, flow schemas, filtering rules, and incident triage instructions. Your engineers run the system independently, or we stay on support with an agreed response time.

Let's discuss the task: what to clarify before we start

To come back with an estimate and a plan rather than generalities, we need input data for your scenario: the set of networks, the required history depth, and the event reaction mode. Below is what most often determines the architecture and stage timelines. Describe the task in free form; a short answer for each item is enough.

• Networks and coverage: how many networks need to be monitored at once, whether any of them are test environments, and whether the list is planned to expand after launch.
• Fixation point: do we react to a transaction appearing in the mempool or only to confirmed blocks, and how many confirmations we consider sufficient.
• Event sources: the list of factories and contract registries, which events determine token birth, and what we consider a duplicate on repeated runs.
• Output composition: which features of interest are needed — liquidity, holder distribution, presence of trading pairs, traces of automated purchases — and which of them are mandatory in the first release.
• Operating mode: notifications only or database writes with historical search, what storage depth is needed, and what to do with incomplete data at an early stage.
• Load and latency: expected number of new tokens per day, acceptable time from event to signal, peak spikes, and system behavior during them.
• Signal delivery and access: where we send them — your interface, a messenger, a call to an external system — who gets access, and which nodes we work on.

With this set, we estimate the scope, propose a processing scheme, and lay out stage timelines: from a prototype on one network to a resilient loop with noise filtering and risk triage.

Write what is already implemented and which areas raise doubts — we will come back with an estimate and a work plan.