Mempool monitoring bot: where transactions are lost and why
The Ethereum mempool is not one shared list of pending transactions but many local queues: each node maintains its own pool with a limited size and its own selection rules.
When the transaction queue overflows, the node evicts the weakest entries, so a mempool monitoring bot connected to a single node sees only its slice of events.
Fee-based eviction works like this: as soon as the pool is full, under pressure from more expensive transactions, an older entry disappears from the queue without ever making it into a block. Formally, it is not rejected — it simply vanished from the stream.
If the bot relies only on a hash appearing in the subscription and does not recheck the final status, the transaction remains recorded as sent, and the associated request hangs.
The second source of loss is a subscription stream break. The connection to the node drops during its restart, network failures, or an endpoint change, and events from the moment of the break are lost silently: the subscription resumes from the current moment.
On the surface, everything works and there are no errors in the logs, but missed transactions accumulate, and the discrepancy surfaces later, when the stream can no longer be restored.
On L2, the picture is more complex: its own admission queue, its own fee thresholds, and its own delay before inclusion in a block, so the same transaction behaves differently in the Ethereum mempool and on L2.
Next, we break down how this is addressed — what to recheck, how to survive breaks, and how to distinguish a lost transaction from a merely slow one.
What continuous monitoring of the transaction queue provides
An event in the transaction queue lives for moments: by the time a scheduled handler wakes up, the opportunity is already gone.
Continuous monitoring of the transaction queue turns event reaction latency from tens of seconds into tenths of a second — the signal goes out at the moment a transaction appears, not on the next polling cycle.
The share of missed events only becomes apparent on a continuous stream. With polling at intervals, some transactions do not make it into any snapshot, and such losses are invisible in the logs: there is simply nothing to record.
Event-driven ingestion with processing acknowledgment and redelivery keeps the miss rate at zero — every delivery failure is recorded and investigated.
Recovery time after a failure is measured not by the process starting but by returning to data completeness.
Automatic reconnection to the node, backfilling missed blocks from a checkpoint, and state reconciliation shorten this path to tens of seconds — instead of hours of blind manual investigation.
A separate effect is the eliminated class of failures. This is not about "the bot crashed" but about a service that runs and stays silent: an event occurred, no signal.
Deduplication, idempotent processing, and correct state rollback during chain reorganization remove this class entirely: a miss becomes a visible incident rather than a silent loss.
Manageability is added: queue metrics, processing lag, retry counter, and alert threshold show that monitoring is alive before this becomes clear from a missed event.
The output is sub-second reaction latency, a near-zero miss rate, recovery within a minute, and no "missed event" class. Every point is verified on a chart, not in words.
Mempool monitoring: operating modes and their limitations
Mempool monitoring begins with choosing a stream observation mode: it determines what the bot is even capable of seeing, and this limitation is primary relative to any filtering logic.
A transaction becomes visible to a node at the moment it is propagated across the network, but appearing in the stream guarantees neither inclusion in a block nor order preservation — some transactions are replaced by price, and some are discarded during chain reorganization.
| Mode | When it fits | Limitations |
|---|---|---|
| Full stream subscription | All events before block inclusion are needed, including unsuccessful ones | Requires a dedicated node with disk and bandwidth headroom; the stream contains duplicates and noise, so filtering falls to the bot |
| Selective address tracking | A limited set of wallets and contracts is of interest | The node returns only matches — context around the transaction is not visible; when addresses change, the rules are rebuilt |
| Scheduled polling | Rare checks, no latency requirements | Events are lost between polls, and activity spikes are missed entirely |
| Confirmed blocks observation | Reliability and reporting are needed, not speed | Data arrives with a delay of several blocks; reacting to an event is impossible |
The modes are not mutually exclusive: a typical setup combines a full subscription for an early signal and confirmed blocks as the source of truth, while selective address tracking reduces load when there are few addresses.
A common mistake is treating the stream as guaranteed: without handling chain reorganizations and transaction replacements, the bot shows events that will not be in the final history.
That is why we record the mode's limitations in the setup description in advance, not after the first discrepancy in the numbers.
How we implement the bot: stages of engineering work
The work proceeds in stages — from stream audit to handover of control. Each step is closed with an artifact that can be checked before going live.
- Stream and node audit. We choose the event source — own node or external provider, confirmation levels, and the method of subscribing to blocks and transactions. We measure peak intensity and the noise share: thresholds, limits, and the need for a backup source depend on this.
- Event processing scheme. We define the transaction route: ingestion, parsing, field normalization, duplicate filtering, queue, handler, notification. We separate the hot path from analytics so that delays do not accumulate during spikes.
- Filter configuration. We assemble rules by addresses, contracts, call methods, gas threshold, and operation type. For L2 — a separate set: there, event order and confirmations differ from the main network. We log every trigger with its reason so the rule can be explained and disabled without changing code.
- Load testing. We run peak stream and connection loss, check whether the bot catches up on what was missed, whether there are duplicates or losses, and how the queue behaves. We separately test switching to a backup source.
- False positive analysis. On a real sample, we adjust thresholds and weights, removing noise without losing significant events. We fix the definition of a false positive so the metric is unambiguous.
- Acceptance and handover. We deliver according to checks described in advance, hand over configurations, the processing scheme, an operations manual, and the rollback procedure. After that — a regulation: how to change filters, update the address list, and what to watch in monitoring.
Configurations, schemes, and regulations that remain with you
We hand over not server access but a kit that lets the bot be deployed, reconfigured, and maintained without us. Configurations, schemes, and regulations live in your repository: an engineer who has never seen the bot before can understand them in one on-call shift.
- Bot configuration files. Separate profiles for the main network and the required L2s: node addresses, size and fee thresholds, lists of tracked addresses and pools. Edits are made via environment variables, without changing code.
- Data flow scheme. The path of an event from subscription to new transactions and blocks through filtering and notification delivery: where the buffer is, where retries are, and how the interval is backfilled after a connection loss with the node.
- Event selection rules. A matrix of conditions: what goes into a notification and what is suppressed as noise — swaps in specific pools, liquidity removal, unusual contract calls, thresholds for your tasks.
- Failure point map. Where the bot loses events: stream break, chain reorganization, queue overflow, notification channel unavailability. For each point — an indicator and the on-call action.
- Notifications. Message templates and routes to a messenger or internal bus, split by criticality — at night only significant events arrive.
- Documentation and regulations. Deployment and updates, on-call procedure, incident checklist, and rollback to the previous configuration.
- Metrics dashboard. Event processing latency, share of discarded noise, number of missed intervals — degradation is visible before it becomes a problem.
Case study: missed event share dropped severalfold
Before implementation, the bot polled the node on a timer once every few seconds: during that time, some transactions made it into a block, and events were lost.
The on-call engineer learned about them from logs after the fact, and balance discrepancies were investigated manually.
Each incident took up to half an hour of reconciliation by addresses and nonces — with a stream of hundreds of transactions per minute, such checks consumed an entire shift.
What we changed
We moved collection from polling to subscribing to the incoming transaction stream of the Ethereum mempool, connected several independent source nodes, and stitched them together with hash-based deduplication.
The filter parses data by addresses, contracts, and method calls instead of pushing everything into the handler.
We separately addressed reprocessing during chain reorganization — previously, that was exactly where events were lost and duplicated. A notification goes straight to the on-call messenger and the service queue; each event has a transaction link and confirmation status.
What the result was
Measurement on the same stream: the missed event share dropped from several percent to fractions of a percent, the time from a transaction appearing in the mempool to notification decreased from tens of seconds to one or two, and manual checks decreased by roughly ten times.
Logs, metrics, and node unavailability alerts are collected in one place: if a source lags or drops, it is visible before an event is lost. The same setup also works for L2 — only the set of sources changes.
Transaction stream or confirmed blocks: which to choose?
Short answer: these are not alternatives but two different data layers. The transaction stream provides early notification before block inclusion, while confirmed blocks provide resilience to chain reorganization.
A bot that listens only to the Ethereum mempool reacts faster but regularly catches events that are not in the final history.
The price of an early signal is noise. A mempool transaction can be evicted, replaced by a version with a higher fee, or fail to make it into a block; during chain reorganization, the observed event disappears from the canonical branch. If the bot executes an action immediately, every discrepancy produces a rollback or duplicate.
Relying only on blocks removes the problem but adds delay: several blocks pass before the required confirmation depth. Therefore, in a working setup both channels live together and differ in role: the stream is the signal, the block is the truth.
In practice, this is double confirmation of an event. The primary signal queues the event and starts preparation, while the action is executed when the same transaction is found in a block at the specified depth.
The hash-based deduplication key, the recent blocks buffer, and replay during reorganization prevent one event from triggering twice.
A separate fork in the road is choosing the data source: your own node provides the full stream and custom filters but requires maintenance; a third-party provider removes the load but limits request frequency and history depth.
For L2, depth verification is configured separately — sequencer and bridge data are confirmed differently.
Doubts and limitations: what to check before starting work
Before starting, it is worth separating two different questions: how the bot behaves under load and what happens when it makes a mistake.
The incoming transaction stream on the public Ethereum network is tens of thousands of events per second, and subscribing "to everything" loads the node more than the bot itself.
That is why selection happens on the node side — by addresses and method signatures, not by brute force in the application — and a separate node is allocated for monitoring, not one shared with other services.
The second class of risks is misses and noise.
A connection break, restart, or node lag creates a "blind window," so the last processed block marker and replay protection are stored on disk: after a restart, the bot catches up on what was missed rather than starting from scratch.
False positives come from canceled and replaced transactions and chain reorganizations — they are filtered out by confirmation count and state recheck.
How much will this load our node?
Selection is performed on the node side, unnecessary subscriptions are not opened, and lag is exposed in monitoring metrics. Under overload, the bot throttles the stream instead of accumulating a queue in memory.
What happens after a restart or crash?
State is stored on disk, so after a restart the bot catches up on missed blocks and continues from the same point. Notifications are idempotent — a repeated alert for the same transaction does not arrive.
How are false positives filtered out?
We check not the fact of appearing in the stream but the final state: replaced transactions, cancellations, and chain reorganizations are discarded. The confirmation threshold is configured for your task — from instant reaction to rollback-protected.
Which limitations remain fundamental?
On layer 2 networks, the stream is formed by the sequencer, and some transactions are simply not visible before block inclusion. We document this so as not to promise a signal earlier than it is physically possible.
What about support after handover?
We hand over the source code, configurations, deployment scheme, and failure response procedure, and we train your team. After that — an agreed support period with a response regulation.
Let's discuss the task and come back with an estimate and plan
To discuss the task concretely, we need input on the infrastructure: which nodes are already running, who holds access to them, whether you listen to the mainnet and L2 mempool, and whether private transaction stream processing is needed.
This determines where the mempool subscription is placed and what event stream the setup can handle without loss.
Send what you have on hand — we will clarify the rest on a call.
- Infrastructure diagram: own nodes or a provider, whether there is an archive node, who owns the keys and access.
- List of networks and L2s in the setup and their priority: what we listen to first and what later.
- What we consider a target event: new transactions, fee replacement, large transfers, contract calls — with trigger thresholds.
- Notification delivery channels and acceptable latency: messenger, HTTP notification, database write, dashboard.
- Expected load: average events per minute, peak spikes, history retention depth.
- Resilience requirements: reconnection after a break, handling chain reorganizations, missed event log.
- Who on the customer side accepts the result and by which metrics we consider the work complete.
We come back with an estimate and work plan: stages, team composition for each stage, timelines, and acceptance points. We will agree on the start order after we align on the event scheme and load.