AI trading bot notification system: where signals are lost
A signal fired in the strategy, the trade opened — but there is no message in Telegram. The delivery chain is short: the bot generates an event, puts it in a queue, a worker sends it to the channel and waits for confirmation.
It breaks at the last step: the Telegram client has a connection timeout, sending fails, no retry is provided — the event simply disappears from the queue.
Notification delay during load peaks turns into a missed signal: while the worker is stuck on one hung channel, subsequent events accumulate and become stale.
A separate category of failures is delivery channel limits. Telegram throttles the message rate during a spike, the mail gateway accepts a batch of emails and silently rejects it, Discord limits the number of calls from one address.
If retries are done "head-on", without an idempotency key, an attempt to resend a message turns into duplicate notifications: the trader receives the same signal three times and stops trusting the channel.
The most unpleasant scenario is a bot restart. The queue lives in the process memory, and everything unsent is erased without a trace when it crashes.
In this case, a sending failure often is not even written to the log: you find out about the loss in the morning, from the discrepancy between the trade history and what arrived in the messenger.
It is exactly at these three points — timeout, limit, restart — that delivery breaks, and they are fixed together. Below we examine how this is addressed.
What fault-tolerant signal delivery provides
A single channel and no delivery confirmations give a delivered-signal share of about 97%: some messages are suppressed by bot limits, some by network timeouts, some by blocking on the recipient side. For trading logic, losing three percent is a missed entry that no one knew about.
Acknowledgement of receipt for each channel turns delivery into a measurable quantity: the log shows not "sent" but "received by the recipient." Retry sending is enabled by timeout or negative response, and a key based on the signal identifier cuts off duplicates — the same position is not opened twice.
We measure delivery time not as an average but by percentiles: the median fits within fractions of a second, and the long tail immediately shows where the channel degrades.
A backup channel picks up the signal if the primary does not confirm receipt within the allotted window — the failure of one service no longer stops the distribution.
The backup role does not have to be given to a second messenger: email covers it no worse while the primary channel recovers, and the channel traversal order is configured for the specific bot logic.
This closes a whole class of failures: the signal is in the log but did not arrive; arrived twice; arrived an hour after entry.
The report leaves three control values — the share of delivered signals per channel, the delivery time distribution, and the number of triggered resends.
Delivery channels: messenger, email, corporate chat
Channels solve different tasks and do not replace each other: a messenger (in the basic setup — Telegram) provides speed of reaction to a signal, email provides the trade log and reporting, and corporate chat provides the workspace of the on-duty shift. The choice is determined not by convenience but by who must react to an event and within what time.
Each has its own limitations, and they define the delivery scheme: message rate limits, bot permissions in a closed environment, delivery delays. Below is a comparison by suitability for a trading system.
| Channel | When it fits | Limitations |
|---|---|---|
| Messenger (bot) | Signals with reaction within minutes: entering a position, protective order | Send rate limit, session break interrupts delivery, receipt is not confirmed |
| Summaries, trade log, incident review | Delivery delay, risk of landing in spam filter, no instant reaction | |
| Corporate chat | The on-duty shift sees the signal in a shared channel, history is preserved | Requires permissions in the workspace and its own event ingestion point |
| Backup channel | Critical events when primary ones are unavailable | Its own authorization, quota control, higher maintenance overhead |
The working scheme is usually hybrid: a critical event goes to the messenger, is duplicated to email for audit, and without confirmation within a given interval is escalated to the backup channel.
Cutting off duplicates by event identifier prevents duplicates from multiplying during resends, and the delivery log shows which channel the notification arrived through. The failure of one channel does not turn into a missed signal.
Implementation stages: from channel audit to handover
A notification in a trading environment is not a chat message but a signal on which a trade depends. If it arrived late or did not arrive, the strategy operated blindly, and incident analysis starts from scratch. That is why the work follows a fixed sequence, where each stage is closed with a verifiable result.
- Delivery audit. We review the connected channels and losses: Telegram rate limits, Email spam filters, Discord restrictions. We divide events into critical and those that can be collected into a digest.
- Routing scheme. We break events down by priority — entering a position, stop, exchange connection error, scheduled report. For each we define the primary channel, backup, and resend rule.
- Channel setup. We separate bots, senders, and permissions, prepare message templates, divide streams by strategies and accounts so that the noise of one does not drown out the signals of another.
- Sending assembly. A queue with delivery confirmation and retries: a connection break does not consume the notification, and a duplicate does not arrive twice. Each message has an identifier and a trail in the log.
- Failure testing. We disable a channel, drive the response into a timeout, stop the sending process, and see what goes to backup. Separately — a burst of events: ten triggers must not turn into a hundred messages.
- Handover to operation. We hand over the route map, templates, and instructions to the on-duty operator. We set alerts on the delivery state itself: the system must report when it has stopped reporting.
After that, the scheme lives its own life: a new channel or strategy is added to the route in one step, without rebuilding the entire environment.
Configurations, routing schemes, and regulations
After implementation, the client's team is left not with a "black box" but with a set of artifacts that can be read, changed, and handed over to another engineer.
Without any item below, the system works, but loses manageability at the first change — a new channel, a change of on-duty operator, or a failure analysis.
- Delivery configuration — channel priorities, trigger thresholds, pause before resend, rate limits per channel and per recipient. Separate profiles for test and production environments.
- Routing scheme — the table "event → channel → recipient group" for signals, trades, drawdown, and data stream break, plus duplicate suppression rules and a quiet period.
- Message templates — markup for Telegram and Discord, text and HTML version of the email, substitution fields, rules for shortening long values.
- Access policies — who edits routes, who only reads the delivery log, who confirms sending to critical channels; roles are tied to accounts.
- On-duty regulations — coverage windows, escalation procedure, events requiring mandatory reaction, and shift handover point.
- Delivery log — records for every sent, deferred, and discarded message with the reason; it shows at which step the notification was lost.
- Recovery instructions — how to mute a noisy channel, move the stream to a backup route, and what to check first if the bot has gone silent.
The set lives together with the system: edits to routes and templates go through the same review process as code. Describe the task — we will return with an estimate and plan, including the documentation scope.
Case study: signal delivery without misses and delays
Before implementation, signals went from the trading module directly to each channel: sending occurred synchronously in the processing loop, during a volatility spike the queue grew, and some messages were discarded by timeout.
There was no deduplication — when the process restarted, some signals were sent again, some were lost. Analyzing each discrepancy between a signal and the actual notification took the team several hours per week.
Delivery was moved to a separate service with a queue and idempotent keys: each event has an identifier, a retry on failure does not create a duplicate, and failed attempts go into retry with increasing pause.
Telegram, Email, and Discord channels are connected as independent routes with their own rules and priorities.
For each message, a record is written to the delivery log with statuses from enqueueing to confirmation, and in Grafana a dashboard is assembled for the delivered share and latency distribution by quantiles.
If a channel is unavailable, the switching rule moves the signal to a backup route, and the event remains in the queue until communication is restored. Loss is possible only if all channels fail simultaneously, and such a case raises a separate alert for the on-duty operator.
Metrics after: median delivery — 1.5–2.5 seconds versus 40–90 seconds before the rework, the share of lost notifications fell from 6–8% to 0.1%, manual investigations decreased from 15–20 per week to 1–2 per month.
Each discrepancy is now raised from the delivery log within minutes: it is visible at which step, in which channel, and for what reason the message did not arrive.
What to do if the bot already sends signals directly
Direct sending inside the trading loop is the most common source of losses.
The synchronous call sits in the same thread where the signal is calculated: the channel responded with a rate limit or timeout — the message went nowhere, there are no retries, and the next signal is already ready. A week later, no one will reconstruct whether there was an entry and why Telegram was silent.
Moving delivery out solves this through separation of responsibilities. The bot logic writes the signal to the outgoing log in the same transaction as the trading event, and a separate process reads the log and distributes messages across channels.
The idempotency key — signal identifier plus channel — guarantees that reprocessing will not duplicate the notification.
It is better to switch to such a scheme gradually, without breaking working code. First, shadow mode: the new path duplicates sending, the old one remains primary, and logs are compared line by line.
Then channels are switched one by one, and the previous sending function remains a wrapper over the queue — backward compatibility is preserved, and strategies and handlers do not need to be edited.
Logging closes the issue of trust in the system. For each attempt, we write the status, time, failure reason, and retry number; undelivered items go into a separate queue with an increasing pause between attempts.
If the loss share exceeds the threshold, the on-duty operator receives an alert — we learn about a silent bot from a metric, not from the user.
Common technical questions about signal delivery
Signal delivery is not "send text to chat" but a queue with confirmation.
The notification module runs as a separate process: the trading core publishes an event and does not wait for a response from an external service, so channel unavailability does not slow down strategy logic.
Each signal is assigned a sequence number and event identifier — a resend does not turn into a duplicate, and the recipient discards what has already been seen. Permissions are configured separately: the sender can only publish, without reading history or access to positions.
How do you know that the signal actually arrived?
The message goes through the states "created → sent → delivered", and the log is kept for them. If delivery time goes beyond the usual corridor, the on-duty operator receives a warning, and the critical signal goes through a duplicate channel.
Can a notification crash or slow down the bot?
No: sending is asynchronous and moved outside the trading loop. When the queue overflows, the module degrades — it writes to the log and cuts non-priority messages rather than blocking the core.
How is a channel checked before connection?
We run a synthetic signal through each channel: format, delivery time, behavior during a connection break, and retry after recovery. The output is a report on delays and losses, not "it seems to have arrived."
Who sees what in the channels?
Access is granted to roles: traders receive signals, analytics receive summaries, the administrator receives the delivery log. Bot keys are stored separately from trading keys, are revoked with one command, and do not grant rights to operations.
Let's discuss your signal delivery scheme
To make the estimate substantive rather than "by eye," send us the signal delivery scheme and a list of message types: what the bot sends, in which channel it should end up, and what is considered critical.
Based on this scheme, we assemble the scope of work, estimate the volume, and return with an implementation plan — with stages, dependencies, and checkpoints.
What we review when discussing the task:
- channel map: Telegram, Email, Discord — which type of signal goes where and which channel is primary and which is backup;
- event sources: trading signals, order execution statuses, errors, end-of-session reports;
- delivery rules: priorities, retries on failure, duplicate protection, quiet hours;
- notification templates and fields: instrument, direction, entry level, and protective orders;
- access and secrets: how bots, tokens, and mailboxes are issued, where keys are stored;
- delivery monitoring: how we find out that a message did not go out, and who reacts first;
- rollback scheme: what happens when the primary channel is unavailable and how we switch.
The scheme can be provided as text, a table, or screenshots of the current dashboard — logic matters to us, not format. Describe the task and attach what you already have: we will return with a work estimate and implementation plan.
