Why a notification system for traders fails under peak load
In quiet mode, events from the trading core arrive one at a time, and the trader notification system manages to send them without delays.
Peak load on the trading core breaks this picture: at market open or on a sharp move, hundreds of stop triggers, margin changes, and order confirmations appear within seconds.
A gap emerges between event generation and delivery to the trader, and it grows faster than the stream itself.
Event delivery latency grows nonlinearly: while the queue is short, it is unnoticeable, but when the backlog overflows, it starts dropping the tail.
Hence a lost account event — the trader does not learn about a triggered stop or a change in maintenance margin and makes a decision based on an outdated picture. A missing notification is more dangerous than a late one: no one notices it.
The opposite extreme is duplicate alerts. If delivery is not confirmed, the retry goes out again, and without an idempotency key the trader receives three messages about one trigger.
The channel starts to be perceived as noise, and it gets turned off — along with the signals that actually matter.
Both extremes grow from the same root: the delivery layer does not know the state of the event. We break down how this is addressed — from the event lifecycle in the core to delivery confirmation at the trader.
What a managed event delivery loop gives the trader
End-to-end delivery latency is measured from the moment of the event on the venue side to the notification arriving on the trader's device, not from entry into our queue.
The path is broken into segments — ingestion, routing, sending to the communication channel — and each is measured separately.
This makes it visible exactly where fractions of a second are lost, and the share of critical alerts arriving within the target window stops dropping under peak load.
A separate class of failures is closed, in which the trader receives three identical notifications for one condition trigger. Each event carries an idempotency key, and a repeat during resend is recognized and is not sent for delivery.
Alert deduplication works at the loop level, not in the client application — this removes both the irritation from duplicates and the false impression that several different conditions triggered.
Event prioritization separates streams: execution, stops, and margin risks go on a separate lane and do not queue behind market summaries. Scheduled mass sends physically cannot displace a single critical event.
Queue observability provides numbers before complaints: depth, age of the oldest message, share of undelivered, number of triggered priority rules. The threshold for an on-call signal is configured so that latency growth is noticed before the trader writes about it.
Delivery modes: instant, batch, and scheduled
The delivery mode is determined not by the convenience of the channel, but by how long the event retains value.
A level breakout, position liquidation, partial execution of a large order lose meaning within seconds — such events are sent instantly, as a separate message to each subscriber.
Reports and period summaries do not require such urgency: it is more efficient to accumulate them and send them in a batch or on a schedule.
Delivery channels are an independent layer on top of the mode: an in-app notification, a call to the trading terminal, a messenger, an email.
The channel changes without restructuring event logic, while the mode dictates requirements for the queue, deduplication, and message order.
Comparison of modes
| Mode | When it fits | Mode limitations |
|---|---|---|
| Instant event delivery | Level breakout, liquidation, trade status change, signal on a liquid instrument | In a volatile market the stream grows manyfold: deduplication and a per-subscriber limit are needed, otherwise the channel turns into noise |
| Batch sending | Series of similar events: mass liquidation, batch of executed orders, portfolio changes | Delay for the accumulation window; order within the batch is not guaranteed, some events arrive outdated |
| Scheduled polling | Summaries, reports, state checks for low-liquidity instruments | Delay by the polling interval; with rare events, some checks are wasted |
The modes are not mutually exclusive: in one notification system they coexist, and routing is set by rules — event type, instrument liquidity, recipient priority.
Then urgent items go out immediately, while background ones do not drown out the channel and do not prevent the trader from seeing what matters most.
Implementation stages: from event analysis to handover to operations
Work proceeds strictly sequentially: routing that has not been tested on a real stream breaks on the first volatility spike. First we record which events exist at all and at what frequency they arrive, and only then design delivery rules. Below is the order in which we run the project.
- Event stream audit. We inventory sources: prices and order book, order execution, risk-limit triggers, deposit and withdrawal statuses. For each, we record frequency, latency relative to the exchange, and criticality — the result is a map of what deserves a notification and what creates noise.
- Routing schema. We determine which event is addressed to which trader, through which channel it is delivered, and with what priority. This also includes aggregation of similar signals, quiet windows for low priorities, and deduplication, so that one market move does not arrive as three messages.
- Assembly and configuration. We bring up the queue, retries on channel failure, delayed delivery, and quarantine for unprocessed messages. Idempotency keys prevent reprocessing from turning into a duplicate at the client.
- Load testing. We run the profile of a real trading day with peak spikes. We look at latency from event occurrence to delivery, the share of lost messages, and order preservation under overload.
- Fault tolerance. We check behavior when a channel fails: switching to a backup, absence of losses, metrics, and on-call shift alerts.
- Pilot operation. We launch on part of the audience and verify with traders what happened in the market and what reached them — at this stage, changes to rules cost the least.
- Handover to operations. We deliver the routing schema, configurations, on-call and escalation regulations, and the procedure for adding new events; we support the first shift and record responsible parties.
What remains with the team: configurations, schemas, and regulations
The project is closed not by demonstrating working alerts, but by the customer's team changing delivery rules itself, adding channels, and investigating failures. Therefore, along with the code, we hand over the configurations and documents by which the system lives without us. The set is recorded in the acceptance act — below is what is checked line by line.
- Handed-over configurations: descriptions of the delivery service, queues, templates, and schedules in the code repository, with environment separation and no manual edits on the server.
- Event flow map: the path from the quote source and trader rule trigger to a specific channel, with retry points, duplicate cutoff, and priorities during peak load.
- Access policies: administrator, support, and trader roles, rights to templates and external channel keys, change log.
- Delivery documentation: message formats, retry rules, channel provider limits, and system behavior on non-delivery.
- Incident procedure: the procedure for diagnosing channel degradation, switching to a backup route, escalation criteria, and post-recovery review.
- Monitoring: metrics and dashboards — delivered share, latency, queue length, errors by channel.
We hand everything over to the customer's repository, with deployment instructions and on-call shift training. After a month of support, it is clear whether the artifacts are workable: rules change without us, and incidents are closed according to the procedure.
Case study: how we reduced losses at peak delivery volumes
A brokerage platform came to us with a working notification service: price alerts, margin calls, order status changes.
While the market was flat, everything looked fine, but during volatility spikes events accumulated in the queue — the trader learned about a protective order trigger after the trade.
The analysis identified three causes. Delivery went as a single stream through webhooks without processing confirmation: some requests timed out and were not retried. Retries went without an idempotency key, so the same alert arrived twice. The third cause was a shared queue: a margin call waited in it behind a mass send.
What the metrics showed
| Metric | Before | After |
|---|---|---|
| Share of lost events | 3.8% | 0.04% |
| Median delivery latency | 9 seconds | 0.6 seconds |
| Maximum latency at peak | about 6 minutes | 25 seconds |
| Share of duplicates | 12% | 0.2% |
| Support requests on a peak day | 380 | 30 |
We separated the streams: margin calls and stop orders went into a priority queue, mass sends into a separate one, with rate limiting.
Each event received an idempotency key, and the recipient received deduplication by it; undelivered items go into a retry with increasing delay and into a log, from which the alert can be sent pointwise. The app, email, and Telegram work through one delivery loop.
A separate effect: support stopped being a delivery channel: requests like “I didn't receive an alert” on peak days dropped by an order of magnitude. New event types are added without rewriting — a new topic and template, with shared delivery and retries.
How does an event stream differ from periodic account polling?
An event stream delivers a notification when the change has already been recorded in a block and has accumulated confirmations. Periodic polling works the opposite way: the service itself requests account state at equal intervals and compares the snapshot with the previous one.
The difference is visible under load. Polling creates a constant background of requests, even when the trader is not making trades: the trading core and the nodes receiving the requests suffer.
The delay here is no less than the polling interval — the execution signal arrives with a lag, and some scenarios (partial executions, an order ladder) are processed after the price has already moved.
The choice between approaches is not either-or, but a question of which channel to consider primary. Below is how they compare in practice.
Completeness of the picture is the second fork. Polling shows only the final state, while intermediate steps (partial execution, cancellation, internal transfer) are not included.
An event stream provides a continuous log: the entire account history can be restored from it, and this also addresses channel interruption.
A channel interruption alone is not survived by the stream: after reconnection, catch-up via the event log is needed, otherwise a gap remains in the history and the trader does not see some trades.
The block log in Ethereum stores all state changes, so a missed interval is restored without losses. Therefore, in practice, subscription is made the primary channel, and polling is left as integrity control.
Technical questions: delivery reliability and event ordering
Order and delivery reliability are solved at the transport level, not at the moment of sending.
Each event receives a monotonic number within the trader's stream and an idempotency key: a resend does not create a duplicate, and a gap is restored by the number of the last confirmed event.
The queue is divided by priorities: a stop order trigger and a margin call go outside the shared line, while analytics and reports go in the background, so that peak load does not shift urgent signals.
Deferred events are stored with a trigger timestamp and are raised into the queue at the appointed moment.
What happens if the trader loses internet?
The event remains in the queue and is retried with increasing delay until the device confirms receipt. After connectivity returns, the client loads the missed segment by numbers — the trader sees the full picture, not just the last message.
The trader logged in from a phone and a laptop — will everything arrive twice?
No. Each device is a separate subscription with its own set of channels: urgent signals go everywhere, reports and digests only to the primary one. Event order is shared, determined by stream numbers.
Can a notification be postponed to a specific time?
Yes, the event is placed with a trigger timestamp and is raised into the queue at the appointed moment. If the condition has lost meaning by then — the position is closed, the order is canceled — the event is removed and does not go to the trader.
Let's discuss the task: where we start and what we need from you
The first task review is one conversation in which we go through the entire chain: where events come from, which delivery channels are needed, what is considered critical, and what can be postponed.
The outcome makes clear the scope of work and what data we need from you to come back with an estimate and an implementation plan, rather than generalities.
So that the scope estimate is based on facts, not guesswork, input data is needed:
- event sources — exchange streams, internal services, brokerage systems, databases — and in what form each provides data;
- composition of notifications and delivery channels: which events go where and whether the recipient chooses the channel themselves;
- prioritization rules — what to do during an event storm, what to merge, what to suppress on repeat;
- reliability requirements: acceptable delivery latency, behavior when a channel is unavailable, whether a retry after failure is needed;
- access to a test environment — a replica of streams or an isolated environment, to test scenarios against live behavior;
- logging requirements: who and how should see whether a notification was sent or not;
- contacts on the business and operations side, with whom we agree on texts and rules.
We break the implementation plan into stages — event assembly and acceptance, routing, delivery, observability — and for each we show what is checked and what confirms readiness.
We name the team composition for the task: an integration engineer, a service developer, a load and reliability specialist. Describe the task — we will come back with an estimate and a work plan.