Crypto payment webhooks: where the event chain breaks
The event chain from the network to the accounting system most often breaks not on the blockchain side but on the recipient side. The wallet records transaction confirmation, the watcher service generates an event and sends an HTTP callback to the merchant's address.
If the address is unavailable at that moment, responds with an error or timeout, the event goes into retries, and after exhausting them it settles in the unprocessed queue.
For the business, this looks the same: the funds are in the wallet, but the order in the system is not paid.
Redeliveries are the second failure point. The sender, without waiting for delivery confirmation, repeats the request; a receiver without an idempotency key treats it as a new event.
This leads to duplicated credits, repeated emails to the customer, and two entries in accounting for one transaction. This is closed by the combination of "a unique event identifier plus an atomic record of payment status together with side effects".
The third problem is diverging statuses. While transaction confirmation is in progress, the merchant already has its own scale: created, pending, paid, canceled.
A chain reorganization or transaction replacement rolls confirmation back, while the internal status remains "paid". Here you need a single source of truth and recalculation of status by confirmation depth, not by the fact of the first notification.
A separate layer is reconciliation: periodic polling over the list of stuck payments closes events that never arrived and reveals discrepancies between the network and internal accounting.
We will break down how this is closed—from a queue with redelivery to an idempotent receiver and discrepancy control.
What changes after implementation: speed, completeness, control
While notifications are based on periodic network polling, between transaction confirmation in a block and crediting, from several minutes to an hour passes.
All disputed requests fall into this window: the client sees the transfer in the block explorer, but the payment has not yet appeared in the accounting system. This is the most expensive class of failures—the funds have already left, but the order is not placed.
Reliable event delivery is built on a queue with delivery confirmation: the recipient's webhook address is called again with an increasing delay, and each attempt is written to the log with the response code.
The share of delivered notifications ceases to depend on the availability of a single service—with retry deliveries it reaches 99.9% versus 90-95% with a single call.
Idempotency of processing is mandatory: events arrive out of order and may be duplicated, so each carries a key—the transaction hash and the event index in the block. The recipient uses this key to filter out a repeat, and redelivery does not turn into a double credit.
Payment reaction time is reduced to seconds after confirmation—instead of waiting for the next reconciliation cycle.
Payment monitoring covers the rest: an alert for a transaction stuck without confirmations, for an increase in reorg depth, for a growing queue of undelivered events. We select confirmation depth for the specific network—for Ethereum this is a separate finality parameter.
As a result, the class of failures "the payment went through, but the system did not learn about it" disappears entirely.
Ways to receive notifications about incoming payments
The mode of payment notification delivery is determined by what is available in the customer's environment: outbound connections, a public address for receiving calls, isolation requirements.
In practice, the choice comes down to three options—periodic polling, an event stream from the node, and webhook calls from the payment service.
| Delivery mode | When it fits | Limitations |
|---|---|---|
| Scheduled periodic polling | Closed environment without inbound traffic; dozens of addresses; reaction in minutes is acceptable | Payment detection delay equals the polling interval; with frequent requests, load on the node grows; history has to be read in batches |
| Event stream from the node | Reaction within a block is needed; high operation frequency | Holds a persistent connection and reconnection; if the connection breaks, some events are lost, and reconciliation by block height is needed |
| Webhook calls from the service | Public receiver; medium and high throughput; several networks | Delivery with retries—the handler must be idempotent; delivery confirmation, a retry queue, and signature verification are needed |
What all modes have in common
None of the options guarantees exactly-once delivery: notifications arrive at least once. Therefore, deduplication by transaction identifier and accounting for confirmation depth are built in from the start, regardless of the chosen channel.
We always add reconciliation polling on top of the active channel—it closes connection gaps and picks up payments for which the call did not arrive.
How we implement crypto payment webhooks: work stages
Sending a notification directly from the payment handler is the fast path and the most fragile: if the customer's service is unavailable, the event is lost, and the payment remains without status.
That is why we work from an audit of the payment scheme to handover into operation, testing each branch against failure scenarios.
- Payment scheme audit. We examine how payment notifications currently arrive, where order state is stored, and what checks are performed before crediting. We record scenarios: redelivery, event loss, block rollback in the network, manual transfer outside the system.
- Notification contract design. We agree on the set of fields, request signature with a secret key, event identifier for idempotency, and versioning—so that adding new fields does not break already working recipients.
- Queue and retry delivery setup. Notifications go through a queue with processing confirmation, retries go with increasing delay, and undelivered ones go to a separate branch. The customer's service can be down; the event will wait for it.
- Test environment. We set up a network emulator with controllable conditions: confirmation delay, block rollback, duplicate event, recipient unavailability. We run each branch and compare logs on both sides.
- Implementation and acceptance. We write code, status storage, metrics, and alerts about queue congestion. During acceptance, we analyze incidents together with your team.
- Handover into operation. We transfer configurations, the flow diagram, and instructions for manually resending an event; we record a rollback plan if the recipient update went unsuccessfully.
The set of settings and documentation that remains with you
Development ends not with a demonstration of a working notification but with the transfer of a complete set of artifacts to your team. Everything is in your repository and on your servers: you can view, modify, or roll back the configuration without our involvement.
- Flow diagram. The payment path from confirmation in the network to the webhook on your server: where the timeout is set, how retry deliveries are arranged, what happens when the receiver is unavailable, and how duplicates are filtered out.
- Handler configurations. Event subscriptions, status mapping, signature keys, retry thresholds—as separate files, not "configured by an engineer in a personal dashboard".
- Integration documentation. Notification body format, field composition, expected response codes, behavior on redelivery, and request examples for the test environment.
- Access rules. Roles and permissions for the panel, secret rotation procedure, storage locations for signature keys, and who is authorized to reissue them.
- Incident procedure. What to check when a notification is missed, how to resend a specific event, and to whom and in what order to escalate.
- Instructions for the team. Analysis of typical failures and a set of acceptance checks that are run after any changes on your side.
Each item is designed for an engineer who will open it six months after launch and figure it out without correspondence with us.
Case study: how the payment flow stopped losing events
Situation before: the payment service accepted notifications about incoming transfers on the Tron network "on the fly"—one handler, without saving the event to disk.
If the service restarted during a rollout or could not keep up under peak load, the notification disappeared without a trace, although the transaction in the network was already confirmed.
Every morning, the operator reconciled statements manually and completed credits with a script by address—some payments reached the client with a delay of a day or more.
What was changed in the delivery scheme
The notification stopped being processed "at the moment it arrives": now it is first recorded in a queue with disk persistence, and only after that is it confirmed to the sender.
The handler is idempotent by transaction identifier—redelivery of the same event does not double the balance. Undelivered items are handled by a background pass with increasing delay between attempts, not by an infinite loop at the moment of failure.
A delivery log was added to this: for each event, the status is visible—accepted, processed, retrying, needs attention. On the payment service side, a stub receiver appeared, so a failure of the main handler no longer creates a data gap.
What changed in the metrics
The share of payments credited automatically grew from 92% to 99.9%. Lost events—from about one per 400 to zero over six months of observation with continuous rollouts.
Manual reconciliation decreased from several hours daily to a short log review once a week: only events with the status "needs attention" have to be checked, and there are just a few of them.
Three classes of failures that previously surfaced only in customer complaints are gone: service restart during load, handler failure on a single "heavy" event, and notifications arriving out of order.
Maximum crediting delay is now measured in minutes and visible in the log, rather than being discovered after the fact.
What to do if payment acceptance is already set up but events are being lost?
If acceptance works on periodic network polling, events are lost not because of code but because of the scheme: the system learns about a transfer only on the next iteration, and between them there is a window when the payment has gone through but for the business it is not there. The wider the list of addresses, the less often each of them gets polled.
Event notifications—webhooks—change the model: the network itself reports the transfer, we verify the signature, and pass the record onward. The difference is not in convenience but in the class of failures, so it is easier to summarize it in a table.
| What we compare | Scheduled polling | Events |
|---|---|---|
| Crediting delay | set by the polling interval | seconds after confirmation |
| Missed on restart | the window between iterations is lost | retry with delivery log |
| Node load | grows with the number of addresses | by the number of new transfers |
| Duplicates | possible when windows overlap | filtered out by idempotency key |
| Block rollback | credit must be reversed | wait for the number of confirmations |
There is no need to switch in one day: polling continues to work, and next to it an event receiver is set up that writes to the same environment through an adapter with an idempotency key—the transfer hash plus the output index.
The accounting system does not change: it receives the usual record or a duplicate, which the adapter filters out. Rolling back the migration means turning off one module, not sorting out completed credits.
Next, we close two residual risks. Undelivered messages—an outbound queue with retries and increasing delay plus periodic reconciliation with the network, which catches what did not arrive. Chain reorganization—by confirmations: we credit after the required number of blocks. The logic is the same for networks such as Tron and EVM-compatible ones.
Common engineer questions about event delivery
In a technical review before launch, the most common questions are about notification authenticity, event order, and system behavior when the recipient's handler is unavailable.
We build delivery so that it does not depend on the receiver's readiness: the event is stored until confirmation, retries go at increasing intervals, and the state of each notification is visible in the log.
How does the recipient know the notification has not been forged?
Each notification is signed with a secret key that only we and the recipient have. The signature arrives in the header and is calculated from the original request body, so it cannot be recalculated from modified data.
On the receiver, the signature is compared in a constant-time manner, and the request is rejected if it does not match.
In what order do events arrive?
Order is not guaranteed: confirmation of an operation may overtake its creation, and delays on the recipient side add desynchronization.
In the notification, we pass the block number, the event position in the block, and a unique identifier—using them, the sequence is reconstructed on the receiver. We make processing idempotent: status change does not depend on whether the previous event arrived before it.
What to do if our service is unavailable?
The notification is not lost: it remains in the queue until confirmation from the recipient, and retries go with increasing delay and jitter so as not to hit the down service in one wave.
When attempts are exhausted, the event goes to the analysis queue: the reason for failure is visible, and sending can be retried manually after recovery. Reprocessing is safe—duplicates are filtered out by event identifier.
Let's discuss the task and come back with an estimate and work plan
To come back with a project estimate rather than clarifying questions, it is enough for us to understand your payment acceptance environment and how crypto payment webhook delivery is set up on your side. Describe the task in free form—we will find out what is missing ourselves.
- How payments are accepted now: where the address or invoice is generated, who records the receipt, and in which system accounting is maintained.
- Which events the recipient needs: crediting, partial payment, overpayment, cancellation, repeated posting after a network reorganization.
- Delivery requirements: how many attempts, whether event order matters, what to do with duplicates, and how notification receipt is confirmed.
- Where we send: the receiving endpoint on your side, queue, accounting or warehouse system, internal services, and those responsible for reaction.
- What already exists: a custom handler, ready-made modules, manual reconciliation—this directly affects the scope of work and stages.
- Volumes and peaks: how many operations pass per day and what spikes the system must withstand without losing events.
- Environment constraints: closed environment, security requirements, access, test bench, and the procedure for deploying changes.
Based on these inputs, we put together a project estimate: scope of work, stages, timelines, and what will be required from your side.
Next—discussion of the task and a work plan: describe how the process is set up now and what result is needed, and we will come back with an estimate and implementation proposal.