Why a new listings screener breaks on the data stream
A new listings screener breaks not on keyword search, but on the data stream.
The venue quietly changes the markup of the announcement page, adds a recommended pairs block, or switches the list to lazy loading — and the parsing rule that extracted the ticker yesterday now returns an empty string.
A missed listing event does not raise an error: the collector honestly fetches the pages, while normalization silently discards everything that did not match the template, and in the logs it looks like a quiet day with no news.
The second bottleneck is time. Cryptocurrency listing monitoring is built on periodic polling, and alert delay is made up of the polling interval, lag of the intermediate storage, and the send queue.
The announcement first appears in the announcements section on Binance, later duplicated in the venue's blog and on social media; if you monitor only one source, the team misses the first minutes of the pair's movement, and that is precisely where the bulk of the price is formed.
Third is event reconciliation. The same listing comes from several sources with different headlines and publication times, and without merging by the 'asset — venue' pair, duplicate events appear and devalue the signal.
Worse is the opposite case: the system is silent, and the team learns about the listing from chat — meaning some sources are not covered by polling, and the failure is visible neither in logs nor in metrics.
We break down how this is addressed: which failure points we check and what we put in place for each, so that misses and duplicates are caught before a human notices them.
What early listing detection delivers: measurable effects
A listing event lives in a short window: from announcement publication to trading start, minutes pass, less often hours.
Everything the team manages to do inside this window — evaluate the project, calculate size, check risks — hinges on one parameter: how quickly the notification arrived.
Early listing detection compresses the time from announcement to decision from tens of minutes of manually browsing exchange websites and channels to a few seconds, and reaction speed to an event stops depending on which on-duty person opens the page first.
Under the hood, this is several independent collection layers: subscriptions to announcement channels and exchange pages, event streams from trading interfaces, and periodic polling of trading statuses.
Each source maps data to a unified event schema — venue, ticker, instrument type, announcement time, start time — after which deduplication by composite key and freshness control are triggered.
Large venues like Binance publish the same announcement through several channels with a discrepancy of seconds, and without normalization the team receives three notifications for one event.
The second stage — scoring — reduces false positives. First, hard cutoff rules: test announcements, delistings, repeat publications, instruments without liquidity and without a pair to a traded base.
Then weighted scoring by features: announcement type, contract availability, venue history, price behavior outside trading. 20–30% of the source stream goes into notifications, and the share of empty signals falls so much that the on-duty person stops turning off the channel.
A separate class of failures that used to cost the most is eliminated: a complete missed event when one source goes down. A backup source, a queue with guaranteed delivery, and a latency metric at each layer turn 'we didn't notice' into a measurable 'processed in seconds'.
Listing monitoring options: comparison of approaches
Venues disclose new listings in different ways: some have an open announcement feed, others only a trading page, and others send the announcement to a newsletter or community chat.
Therefore, comparing approaches starts not with 'which method is better,' but with the question of where the primary source appears and what delay we consider acceptable.
| Option | When it fits | Limitations |
|---|---|---|
| Scheduled polling | Venues without an open event feed | Delay equals the polling interval, extra requests, missed withdrawn announcements |
| Streaming subscription | The venue itself emits events, first minutes matter | Dependence on channel stability, need to backfill missed events |
| External market data aggregator | Broad venue coverage and cross-checking are needed | Intermediary delay, incomplete fields, discrepancy with the primary source |
| Manual browsing of channels | Closed announcements and non-standard venues | Does not scale, depends on a person, no guarantee of completeness |
How this works together in practice
Event streaming versus polling is not an either/or but a division of roles: streaming gives speed, polling covers venues without subscriptions and insures against misses.
We keep an external market data aggregator as an independent cross-check loop — a discrepancy with the primary source is often the earliest signal.
We deduplicate incoming events by listing identifier and first announcement time; otherwise one listing arrives in alerts three times.
How we build the monitoring pipeline: stages of work
The screener is assembled in stages, and the first step is not code but inventory: where the venue publishes announcements, with what delay they appear, in what form they arrive, and what happens when one is missed.
Before implementing the screener, we record the list of sources, event frequency, and acceptable reaction time — then the event processing pipeline is designed for the real stream, not the market average.
- Task analysis and source inventory. We identify the venues in scope, announcement channels, typical delays, and failures of each source.
- Venue data normalization. A unified directory of tickers and names, mapping pair types, dates, and statuses to one schema — events become comparable with each other.
- Deduplication and merging. One listing arrives from several channels: we match duplicates and link the announcement to an already known instrument.
- Scoring threshold configuration. We move feature weights and trigger boundaries into configuration — they can be edited without rebuilding or restarting the service.
- Historical validation. We run the rules against accumulated events, calculate the share of true and false signals, and fine-tune thresholds before going live.
- Alert delivery. Priorities, grouping, and protection against a flood of same-type signals so the on-duty person sees the message in their channel.
- Handover of maintenance procedures. Instructions for connecting a source, editing a threshold, and troubleshooting incidents, plus pipeline health metrics.
What the client receives: artifacts and documentation
The delivery is structured so the system can be operated and developed without us.
We hand over not only a working service, but also settings, evidence of their correctness, and documents by which an on-duty engineer or analyst can make changes without digging through the entire codebase.
- Environment configurations. Deployment manifests, environment variables, secrets in secure storage — the environment is brought up from the repository with one command, without manual edits on the server.
- Source map. List of exchanges and aggregators, polling order, distribution of limits across processes, behavior when a source is unavailable: pause, retry, switch to backup.
- Scoring rules. Feature weights, thresholds, aggregation formula, rule versioning, and change log — rollback to the previous revision without rebuilding the service.
- Alert routing matrix. Signal class, recipient, channel, priority, quiet window, and escalation if the signal is not confirmed. The table is edited by you, not hidden in code.
- Operations procedures. Step-by-step instructions for typical failures, on-call schedule, rate limiters, and safeguards against a flood of false positives.
- Documentation. Architecture diagram, data model, description of external integrations, and an analyst guide: how to add a source or change a threshold.
- Acceptance scenarios. A set of reference listings with expected scores — on the next rule change, it is immediately clear what exactly broke.
Case study: listing response time dropped from hours to minutes
Before implementation
The on-duty shift manually went through exchange feeds, announcement pages, and project channels — one verification round took 20–30 minutes, so listings were discovered 40–90 minutes after publication.
Some events arrived during off-hours and were lost, some were duplicated from different sources, and for a new instrument there were not enough features for a quick decision.
The share of missed listings stayed around 15%, and about the same amount of time went into sorting through noise.
What was changed
The event stream was assembled from exchange feeds and public aggregators — quotes and metadata were pulled via CoinGecko — and mapped to a unified schema: instrument, venue, publication time, link to the primary source.
Deduplication by the 'instrument + venue' pair removed repeat alerts, and a priority queue kept events from getting lost during peak load. Scoring calculated features separately, so the threshold for each channel was configured without rewriting the logic.
What the result was
- Median event detection: from 47 minutes to 1.5 minutes, including overnight windows.
- Share of missed listings: about 15% → less than 2%, every miss is visible in the log.
- Manual work of the on-duty shift: down by about 70% — the operator confirms the event instead of searching for it.
- False positives after threshold calibration: reduced fourfold.
How does streaming monitoring differ from scheduled polling?
Streaming and polling solve one task — learning about a listing before the market — but they pay for it in different ways. A streaming subscription delivers the event at the moment of publication: a persistent connection keeps latency in seconds and preserves announcement order.
Scheduled polling gives even and predictable load, but the time between passes is a blind spot, and a spike of listings within it is lost.
The fork is determined by event frequency and the cost of being late. Where a venue publishes listings in batches, streaming wins: polling once a minute will see the fact, but will miss the order and the first spike when the price has already moved.
On rare exchanges with a couple of announcements per day, a persistent connection is excessive — its breaks have to be restored and missed events backfilled separately.
Therefore, in practice, a hybrid event collection scheme is built: streaming is the main delivery channel, polling is the safety net.
It periodically reconciles the entire listing directory, closes windows after a connection break, and catches events that were not in the subscription.
Reconciliation is done by instrument identifier and publication timestamp, so a repeated arrival of the same event does not create a second notification.
The mode is chosen by one criterion: how long the signal remains valuable. If it lives for minutes, polling as the only source is not suitable — you need streaming with fast subscription recovery.
If the event remains relevant for hours, polling is enough, and there is no need to complicate the scheme. The hybrid provides one more property — degradation without blindness: if streaming fails, collection continues via polling, and such events are marked as less timely.
Technical concerns about scoring and alerts
Most often, before starting, questions come up about thresholds, limits, and system behavior during failures. Below are five scenarios and how they are resolved at the configuration level; each can be tested on a pilot set of sources.
How do you configure sensitivity without drowning in false positives?
The threshold is assembled from feature weights: pool age, volume, holder distribution. For the first weeks, the system runs in observation mode — the threshold is calibrated on the accumulated sample, not on intuition.
What should be done about source rate limits?
Polling goes through a pool of sources and a priority queue; the schedule distributes the load. When hitting a limit, the check moves to a backup source, and critical data is cross-checked across two independent points.
How does the system survive a source or channel failure?
Events are buffered and not lost: undelivered ones are retried, and hash-based deduplication cuts off duplicates. The log shows exactly what was not delivered and when.
How are access rights separated?
Roles are separated: the analyst edits rule weights, the observer sees only output, the administrator manages sources and keys. A threshold change requires confirmation and is recorded in the log.
Can custom alert channels be added?
Transport is separated from scoring logic: a new channel is connected via an adapter, and the core does not change. Filters by event type let part of the team receive only critical items.
Let's discuss the task: what is needed to start work
To start the project, a document dozens of pages long is not needed — initial inputs are enough for us to understand the scope and come back with an estimate and work plan for the screener. Below is what is worth preparing for the conversation; we will clarify some points on a call.
- List of exchanges and networks and the event definition: what we consider a new listing — announcement, trading start, pair appearance.
- Sources: official announcement channels, exchange APIs, on-chain data — and whether access to them exists.
- Scoring rules: which features matter (liquidity, volume, project age, holder distribution) and from what score a signal is considered significant.
- Alert channels and their priority: messenger, webhook to your system, email — who receives the message and in what form.
- Latency requirements: how many seconds from publication to alert are acceptable and what to do when a source is missed.
- History and analytics: for what period to store events and whether a sample of past listings is needed.
- Environment and access: where we deploy, whether a test environment is needed, who is responsible for keys to external services.
With this set, we come back with an estimate and work plan: the sequence of stages, what is included in the first working loop, what quality checks there are, and how you will accept the result. Describe the task — we will start by reviewing the initial inputs.