Lightning Network transactions hang at peak load
When a payment on the Lightning Network does not complete within the allotted window, the invoice in the bot times out — but does not disappear. The user sees “payment failed,” presses the button again, while the first attempt is still hanging in the route at that moment.
This is how a Lightning Network stuck payment arises: the network holds a reserve for the operation, while the interface already offers to repeat it.
Most often the issue is insufficient channel liquidity. Incoming and outgoing balances are skewed: a route participant physically cannot route a payment of the required size, and it repeatedly goes into searching for detours instead of failing fast.
The larger the amount and the smaller the free reserve on the node, the higher the chance that the operation will hang or time out.
The second cause is status desynchronization. The bot tracks state by its own timer, the network by its own, and while the former waits for a response, the payment may already have gone through.
Pressing “pay” again leads to a second charge in the bot, and the user’s balance diverges from the actual network state — reconciliation has to be done manually, using the transaction log.
Add to this a payment invoice timeout set “with a margin”: the user does not understand whether to wait or pay again, and chooses the latter.
That is why payment in the Telegram bot fails specifically at peaks — when liquidity is distributed unevenly and invoices live longer than usual. Next, we look at how this is solved at the architecture level.
What a custodial scheme provides: speed and predictability
A custodial scheme removes from the payment path everything that depends on third-party nodes.
The service holds the keys and channels, so a transfer between two users of the bot is a record in the internal balance ledger: crediting takes fractions of a second, not minutes waiting for confirmation.
Settlement with the Lightning Network moves to the background and happens in batches, after the person has already seen the funds in their balance.
Hence the second effect: the share of successful payments. In the public network, a transfer may fail to find a route with sufficient liquidity, and the user gets a rejection for no apparent reason.
Inside the service there are no routes at all, and outgoing payouts go through our channels, where liquidity is topped up in advance — the success rate of operations stays above 99%.
An entire class of failures disappears — the “stuck payment,” when funds are locked in someone else’s route, the status is unknown, and support has to sort it out manually. An internal transfer is atomic: either crediting or an error before debiting. A state of “in progress” lasting for hours simply does not exist in this scheme.
Recovery time is calculated differently. Channel state and the ledger are restored from the last signed copy — without resynchronizing with the blockchain and without manually sorting out hung operations. After a failure, the node accepts payments again within minutes, not hours.
Predictability of settlements is a consequence of control: fees and liquidity are under our management, mass payouts follow a schedule and do not depend on how loaded the network is at a given moment.
Lightning Network: fund storage modes in the bot
The storage mode determines who is responsible for funds if the bot fails and how expensive support is. Below are four schemes that are actually chosen when launching Lightning payments in a bot, with their limitations.
| Mode | When it fits | Limitations |
|---|---|---|
| Custodial | Mass micropayments and fast launch: the user is not ready to hold the key themselves | Full trust in the operator; it is also the single point of failure and the subject of requirements for storing third-party funds |
| Non-custodial | Large amounts, when the user manages the balance themselves | Access recovery is the user’s responsibility, support is more complex, part of the audience is filtered out at onboarding |
| Hybrid | Turnover goes through the bot, while the reserve is held in the user’s wallet | Dual logic of limits and transfers between perimeters, more complex reconciliation and reporting |
| External provider | No infrastructure of your own, launch is needed without on-call duty | Dependence on the provider’s limits and availability, routing outside your control, payment data goes to a third party |
When control over routes and fees matters, the scheme is deployed on your own node — but that already means operations with channel liquidity monitoring and on-call duty.
Keys and access to funds in any option are described by policy: who on the team can do what, how often keys are rotated, what happens upon compromise, and how access is recovered.
The hybrid is more often chosen as a compromise — turnover is kept small, and the remainder is not stored on the bot side.
Implementation stages: from scenario analysis to handover
The starting stage is not code, but an analysis of how payments actually move in the product: who initiates a transfer, at what moment responsibility passes to the custodial service, what the user sees when the connection between Telegram and the node breaks.
We document these scenarios in writing, because they determine both the channel topology and the requirements for the transaction log, as well as the depth of failure testing.
- Payment scenario audit. We collect all branches: incoming and outgoing transfers, refunds, underpayments, repeated attempts after invoice expiration. The output is a list of bot states and the rules for transitions between them.
- Fund flow scheme. We describe how the user’s internal balance relates to the balance on the node, where keys are stored, and at what moment an operation is considered complete. This scheme becomes the basis of the accounting logic and the transaction ledger.
- Channel and liquidity setup. We determine the number of channels, their distribution by direction, and automatic rebalancing thresholds so that incoming capacity does not run out during peak hours.
- Fault tolerance testing. We run through node loss, provider unavailability, and stale state: recovery from backup, correct channel closure, and handling of stuck payments without double crediting.
- Reconciliation and monitoring. We automate matching of the internal ledger with the node state and set alerts for divergence, capacity shortage, and growth in the queue of unsuccessful operations.
- Handover and support training. We provide configurations, diagrams, instructions for typical incidents, and conduct first-line training: how to read statuses, what to escalate, and which actions are safe without an engineer.
What you keep: configurations, diagrams, procedures
We hand over not access to the bot, but a package with which the infrastructure can be deployed and maintained by your team. Everything is kept in a closed repository with a revision history: you can see who changed Lightning node parameters, limits, and routing rules, and when.
- Bot configuration — node and service settings files, limits for a single payment and daily turnover per user, route selection rules, and fee size.
- Fund flow diagram — a diagram of fund movement between wallets, channels, and the reserve, with the boundary of the custodial perimeter and a list of keys.
- Access policies and roles — a matrix: who withdraws funds, who only views, who changes configuration; second confirmation for critical operations.
- Incident response procedure — what we check when the node is unavailable, a channel is broken, or a transaction is stuck, what we restart, and when we escalate.
- Scheme documentation — description of the architecture, failure points, and the recovery procedure from backup with a verified deployment time.
- Monitoring and alerts — thresholds for channel balances, payment delay, and error rate; notifications go to the on-call channel, not to an engineer’s direct messages.
We finish with a handover session: your team reproduces node recovery on a test perimeter with their own hands and then handles maintenance independently.
Case study: subscription bot with payment growth without channel breaks
Before migration
The subscription bot handled several thousand renewals per day, and each payment went through a separate channel with its own balance.
At peaks, liquidity spread out: the share of successful payments dropped to 62%, invoices expired, and some channels ended up in a state where a payment would not go through in either direction.
Repeat charges were not idempotent — on timeout, a renewal went out as a second invoice, and refunds were handled manually.
What was changed
We built a custodial scheme: a single pool of managed channels and scheduled rebalancing, plus a trigger when the local balance drops below a threshold.
Each renewal received a fixed identifier — a repeated attempt returns the result of the first rather than creating a new charge. At peaks, renewals go through a queue with limited concurrency so that the flow does not consume the liquidity of a single node.
Metrics before and after
- Share of successful payments: 62% → 96% on normal days and no lower than 94% under fourfold peak load.
- Repeat charges: 1.8% → 0.02% of renewals.
- Recovery after node failure: from 40 minutes of manual switching to 3 minutes of automatic failover to the reserve.
The scheme runs on Lightning Network; fault tolerance relies on automation, not an on-call engineer — so subscriber growth no longer runs into manual investigations.
What to do if the bot is already running and payment acceptance is needed
A running bot means a constant stream of requests and a live database, so improving an existing bot begins not with code changes, but with separating perimeters: new payment acceptance is deployed alongside the production one and writes to the same tables, while current scenarios continue to be served.
Adding payment to the bot does not require a pause — old handlers remain enabled while new ones pass validation on real operations of limited volume.
Only the confirmation layer changes: instead of manual reconciliation, a Lightning Network node appears with channels, incoming liquidity, and confirmation tied to a specific invoice.
The bot logic itself — commands, roles, menus, access granting — remains the same, because it does not know where the confirmation came from.
Migration without service downtime proceeds in stages: first, schema expansion with backward compatibility (new fields are optional, old code ignores them), then shadow mode, when payment is accepted but crediting is reconciled with the current mechanism, and only after they match — switching over. Rollback is a single setting that returns the previous confirmation path.
Compatibility with the current database comes down to one question: do you have an internal user identifier that does not depend on the messenger.
If yes, the payment is linked directly to it, and the transaction log, to which records are only added, covers reconciliation and discrepancy analysis with a query instead of reading logs.
We design the custodial part separately: where keys are stored, who initiates a refund, what happens when incoming liquidity is insufficient, and how the bot behaves when the node is unavailable.
These scenarios are tested before payment acceptance is enabled — otherwise the first failure will occur exactly on the first large operation.
Technical questions: liquidity, keys, fault tolerance
Technical doubts around a custodial bot in Lightning Network come down to the boundary of responsibility: what the service controls and what remains on the user’s side. Below are answers to the questions that most often come up before development begins.
Who is responsible for user funds?
The node keys are under the service’s control — hence responsibility for the funds on it. Transaction accounting is kept separately from channel state, balances are reconciled automatically, and any divergence blocks withdrawals until it is investigated.
How are keys protected?
The bulk of funds is kept outside the hot perimeter; the node operates with a limited operational balance. Keys never leave the secure environment, access is role-separated and logged.
What happens when the node is restarted?
Channel state is persisted to disk; after startup, the node restores connections and catches up. Payments during this time are queued and go out after recovery; duplicates are excluded by operation identifier.
What to do when liquidity is insufficient?
Incoming and outgoing liquidity is distributed across channels; when skewed, rebalancing is started or new channels are opened. Thresholds warn in advance, and some payments go via a backup route.
Can the storage model be changed?
A transition to users holding keys themselves is possible, but it means reworking accounting, key issuance, and migration of active channels — we plan it as a separate stage.
Let’s discuss the task: we’ll analyze the scheme and come back with an estimate
To come back with an estimate and work plan, rather than an approximate “plus or minus” range, we need input data from the customer.
The analysis begins with payment scenarios: who tops up the balance, what happens when the connection breaks, and how the bot should behave at that moment.
These answers show which parts of the custodial scheme fall on the server and which remain on the Lightning Network node side.
What to prepare for the discussion:
- Payment scenarios: acceptance, payout, transfer between networks, splitting a large amount into parts.
- Volumes: peak load, daily number of operations, expected growth, average and maximum payment.
- Refund requirements: timeframes, partial refund, disputed operations, manual confirmation by an administrator.
- Access roles: who sees the balance, who initiates a payout, whether second-person confirmations are needed, how the action log is maintained.
- Support constraints: operating mode, acceptable downtime, who on the customer side responds to failures.
- Project handover procedure: source code, infrastructure diagram, instructions for the on-call person, team training.
- Current infrastructure: own node or third-party, which integrations with accounting and reporting are mandatory.
Next, we assemble the scheme and estimate labor costs by stage. If some data is not yet known, we record assumptions and validate them at the first step — so that refund requirements and the project handover procedure do not surface as surprises in the middle of the work. Describe the task in any convenient form, and we will come back with a plan.