Why a push protocol breaks during volatility spikes
When the market moves in one direction, the queue of liquidation notifications grows faster than it can be processed.
A push protocol built around a single sender node starts to accumulate events: first, delivery latency grows from seconds to minutes, then some liquidation messages are lost when the service restarts, and after that duplicates arrive — the same trigger is processed again.
For a position that is being forcibly closed, the difference between "found out immediately" and "found out ten minutes later" is the difference between a managed decision and the fact of losing the collateral.
The cause is almost always the same: the system is designed for average load, not peak load.
Market volatility is not a smooth increase but a surge of dozens of times within minutes: there are more events than the queue's throughput, retries begin to compete with new events, and node replicas diverge in state.
Liquidation monitoring at such a moment turns into a source of noise: some alerts arrive late, some arrive with the wrong priority, and some do not arrive at all.
A separate problem is ordering and idempotency.
If a liquidation event is processed twice, the user receives two notifications about the same thing, and if processing happens in parallel without a deduplication key, the cause-and-effect relationship is lost: the alert about the position's health dropping arrives after the alert about the liquidation itself. At peak, this looks like chaos, even though individually each component works correctly.
We break down such failures along three axes: where exactly the event is lost, what happens to the queue during a spike, and how the system behaves when a single node fails.
Below is how this is addressed at the level of the delivery architecture, rather than through cosmetic edits in the sending code.
What a reliable notification channel gives a DeFi protocol
Speed of reaction to a critical event
A notification about collateral falling below the threshold arrives within seconds, rather than after an operator refreshes the dashboard.
This changes the class of reaction: instead of "we found out 10-15 minutes later," the team gets a window in which liquidation is still manageable — they can top up the collateral, partially close the position, or move it into a protective mode.
In a volatile market, the difference between "we made it in time" and "we didn't" is the difference between a planned operation and forced liquidation with slippage.
Share of delivered events
A channel with delivery confirmation and retries keeps the delivery of critical events at a level close to complete: if the device is offline or the app is closed, the notification arrives via a backup route.
Priority is configured separately — a liquidation event does not end up in the same queue as informational noise and does not get lost behind it. Delivery errors are visible in the logs, rather than being discovered after the fact from an incident review.
Less manual monitoring
The on-call engineer stops keeping charts open and checking position status manually — they react to an event. This relieves the team of routine watch duty and reduces the cost of an error: a person will not miss a signal due to fatigue or switching between tabs. The freed-up time goes into analyzing risks and tuning thresholds, rather than into observation.
Predictable recovery
When the channel fails, events are not lost: they accumulate and are delivered after connectivity is restored, so no missed reaction window occurs.
Recovery time after a failure is measurable — from the moment of the failure to the first delivered notification — and this metric stays in minutes, not hours.
The class of failures narrows down to rare cases, and each of them is analyzed using delivery logs, rather than guesswork.
Event Delivery Methods: Comparison of Operating Modes
A liquidation event needs to reach the user within seconds, otherwise the position will be closed without their participation.
But "delivering a notification" is not a single mechanism, but several modes with different costs of ownership: some provide instant reaction, others provide delivery guarantees, and still others provide minimal coupling with infrastructure. The choice depends on what is more critical for the protocol: speed, reliability, or simplicity.
Below is a comparison of approaches that are realistically chosen between when building monitoring for liquidations and protocol events.
| Delivery option | When it fits | Limitations |
|---|---|---|
| Direct send to channel (push protocol) | Reaction within seconds is needed: liquidation, rate change, threshold trigger | Requires user subscription; if the channel is unavailable, the notification is lost without retry |
| Queue with delivery confirmation | Critical events where receipt matters: loan status, order execution | Confirmation delay; requires tracking the state of each message |
| State polling on request | The user checks the position themselves; background dashboards | Delay until the next request; not suitable for urgent events |
| Webhooks to external systems | Integration with internal services, CRM, team bot | Dependency on recipient availability; retry needed on failure |
| Publication to a shared event stream | Analytics, audit, multiple consumers of the same event | No addressability; the consumer filters what they need themselves |
The key limitation of any mode is not the transport, but the subscription state and retry on failure. Direct send is fast, but does not survive channel unavailability; a queue is reliable, but adds delay.
In practice, modes are combined: an urgent event goes directly, while confirmation and retry are handled by the queue.
When scaling, the bottleneck becomes not the number of channels, but the volume of events per subscriber.
Filtering on the sender side — by address, event type, threshold — reduces load many times over, and separating streams into urgent and informational prevents critical notifications from drowning in noise.
How we implement liquidation monitoring: stages of work
Liquidation monitoring is not "turning on notifications" but building a chain from an event on the network to a message on the user's device so that no critical change is lost.
We start with an audit of the current scheme: we look at which positions are tracked at all, where the data source is, how the loan health threshold is determined, and we find places where the signal may not arrive or may arrive with a delay.
- Scheme audit. We analyze the contracts, oracles, and the collateral ratio calculation logic. We record which events we consider critical: approaching the threshold, the fact of liquidation, rate changes, protocol pause.
- Route design. We determine where we take data from — network logs or our own indexer — and how we link the wallet address to the delivery channel. At this step we decide what goes out instantly and what is aggregated.
- Trigger setup. We define the trigger conditions with a margin on the threshold so that the user receives a warning before liquidation, not after. We eliminate duplicates and false triggers during sharp market fluctuations.
- Delivery check. We test the entire path on real and synthetic events: latency control, resending on failure, correctness of the text and action links. We separately check the behavior when one of the channels is unavailable.
- Documentation. We describe the flow scheme, thresholds, message formats, and the procedure for changing rules — so that the client's team can add new event types on their own.
- Handover to operation. We set up monitoring of the notification system itself: we see how many events were caught, how many were delivered, and where the signal is lost. We hand over access and the incident response procedure.
Each stage ends with a verifiable result, so the launch does not turn into "we turned it on and hope." Describe your scheme — we will get back to you with an estimate and a work plan.
Artifacts and configurations you receive
We hand over not "access to a service" but a working notification loop that your team can maintain without us. Everything described below lives in your repository and your infrastructure, with a change history.
- Flow diagrams: the path of an event from the source (on-chain log, indexer, price oracle) to the user's device — with failure points, retries, and deduplication.
- Filtering rules: which events trigger a send (position health threshold, liquidation size, order status change), which are suppressed as noise, and how thresholds are configured without editing code.
- Message templates: texts and payload structure for liquidation, approaching threshold, order execution, and failures — with localization and variables tied to a specific position.
- Channel configurations: priorities and fallback delivery routes, behavior on permission revocation and on an inactive device, per-user rate limits.
- Operational procedures: what to do during an event spike, how to disable noisy rules, the verification order after changing contracts or a price source.
- Monitoring and alerts for your team: delivery metrics, share of discarded duplicates, latency from event to send, alerts on degradation.
- Documentation and handover: integration description, a checklist for connecting a new event, incident postmortems, and rules for rolling a configuration back to a previous version.
We separately define the boundaries: where our zone ends (delivery and filtering) and yours begins (risk management logic). This removes disputes when investigating why a user did not receive a notification.
Case study: how we sped up liquidation response by 4x
What it was before
The protocol on Push Protocol sent notifications when a position approached the liquidation threshold, but the channel worked like a "broadcast": a single stream for all positions, with no prioritization or deduplication.
During volatility, the event queue swelled, some notifications arrived late, and some were lost when the client reconnected — the operator would only learn about a critical position after the liquidation had already triggered.
Incident analysis was done manually: an engineer would export logs, match transaction hashes against database records, and reconstruct the timeline.
A single incident took several hours, and the share of lost events reached 12% — while tracing a missed event after the fact was nearly impossible.
What we changed
We restructured the event subscription: we split the streams by risk level (approaching the threshold, critical zone, liquidation event) and added delivery confirmation — the client acknowledges receipt, and unconfirmed events go into retry with an increasing interval.
Deduplication by position identifier eliminated duplicates on reconnect, and a buffer on the sender side preserves events during a brief connection loss.
At the same time, we removed the manual analysis: every event is written to a journal with a timestamp and delivery status, and discrepancies between the expected and actual position state are flagged automatically.
What it became
| Metric | Before | After |
|---|---|---|
| Response latency to a critical event | minutes | seconds |
| Share of lost events | up to 12% | below 1% |
| Time for manual incident analysis | hours | minutes |
| Duplicate notifications on reconnect | regular | none |
The speed of response to liquidations increased by 4x, and the operator is no longer a bottleneck: the system itself shows which positions require attention and keeps a full delivery history.
How does push protocol differ from polling monitoring?
Polling monitoring is straightforward: every N seconds the service pings a node or indexer and asks "what's new for my addresses." The simplicity is deceptive — the interval is the delay.
By the time the next request hasn't gone out yet, the event has already happened, but you don't know about it.
For liquidation in DeFi, this is a window of a few seconds during which the position drops below the threshold, and the decision is made not by your service, but by someone else's liquidator.
Push protocol flips the model: a subscription is registered once, and the notification comes from the event source at the moment it appears in a block. The delay is determined not by your timer, but by delivery time — seconds versus the polling interval.
The difference is especially visible on rare events: polling either catches them late, or you shorten the interval and pay for it.
Load is the second side.
With polling, each client generates a stream of requests regardless of whether there are events: a thousand subscribers on an interval of a couple of seconds is a constant background that grows linearly with the number of addresses and doesn't subside in a calm market.
The push model sends traffic only when an event actually occurs, so during quiet periods the infrastructure is almost idle, and during volatile periods it scales based on actual demand, not on a schedule.
Hence the difference in infrastructure cost. Polling requires maintaining a request reserve for the peak interval, meaning you pay for the worst-case scenario around the clock.
Push shifts the main work to the source side and the delivery broker, while your service handles processing — it needs capacity for the real notification stream, not a hypothetical one.
There is also a compromise: a hybrid scheme where push covers critical events — liquidations, large transfers, parameter changes — while polling remains for state reconciliation and recovery after gaps.
This removes the main risk of the push model: if delivery fails, periodic reconciliation restores the current picture.
The choice between approaches is a choice between predictable latency and predictable load; for liquidation monitoring, push with polling as a fallback almost always wins.
Common technical concerns about integration
Integrating a push protocol into a DeFi product almost always comes down to the same questions: how it fits the current stack, what about key and subscription security, whether it will hold up under load during mass liquidations, and whether you can roll back if something goes wrong. Below are the things most often asked during the evaluation stage.
How compatible is the push protocol with our current stack?
The protocol does not require rewriting contracts or changing the network — it connects as a separate delivery layer. On the backend side, we add a sender service that listens for events through an RPC node or indexer and passes them into the channel.
If you already have an indexer (The Graph, your own), it becomes the source of triggers, and no duplicate infrastructure is needed. We verify compatibility on a test network before production.
How is the security of keys and subscriptions ensured?
The sender's private keys are stored in a secret manager or hardware module and never end up in code or the repository. A user's subscription is tied to their wallet — we do not gain access to funds, only to the address for delivery.
We separate channels by permissions: one service sends notifications about liquidations, another about protocol events, and compromising one does not expose the other.
Will the system hold up under load during mass liquidations?
Peak load is exactly the moment when hundreds of positions go into liquidation simultaneously. We place a queue between the event source and sending so that a burst does not turn into an avalanche of requests, and we rate-limit by address.
Prioritization: critical notifications (liquidation risk) go through first, informational ones with a delay. This provides predictable delivery even during peak hours.
What about rollbacks if the integration goes wrong?
The integration is isolated: disabling the sender service does not affect the contracts or the core product. We keep the configuration in the repository, so returning to a previous state is a version switch, not a manual edit.
Delivery logs are retained so we can investigate an incident and understand which notifications went out and which did not.
If your case is not on the list — describe the task, and we will come back with an estimate and an integration plan.
Let's discuss your task and come back with a plan
To come back with an estimate and a plan rather than generalities, we need to understand your architecture and the scenarios in which the push protocol will operate.
Tell us which events are critical: approaching the liquidation threshold, a rate change, order execution, position movement? This determines which subscriptions and filters we build into the contract and how we design delivery.
Send us the source data — we use it to calculate the scope of work and risks:
- protocol description: networks, smart contracts, where position state is stored;
- list of events that require a reaction, and their priority;
- latency requirements: how quickly a notification must reach the user;
- delivery channels: web, mobile app, messengers, e-mail;
- expected load: number of active wallets and events at peak;
- access: test network, repository, contract documentation;
- infrastructure and security constraints, if any.
In response, you will receive a work plan with stages, the composition of subscriptions and filters, a diagram of the event flow from contract to delivery channel, and a timeline estimate.
Separately, we will discuss how we verify reliability: test scenarios for sharp market movements, handling of missed and duplicate events, and rollback in case of delivery failure.
Next — the following steps: we agree on the scope, lock in the stages, and launch the first one. Describe your task — we will come back with an estimate and a plan.