Development of a secure hot wallet for a crypto exchange

What breaks in an exchange's hot wallet under peak load

An exchange's hot wallet is the only circuit where transaction signing and fund withdrawals run synchronously with the client request. Under steady load it looks reliable: the queue is empty, signatures complete in seconds, balances reconcile.

The withdrawal bottleneck shows up at peak — during mass withdrawals or a sharp market move, when a minute brings the volume meant for an hour.

The signing node is the first to fail. If keys live in a single instance of a hardware module or in one process, failure of the signing node stops all withdrawals — not individual operations, but the entire queue.

Delayed retries do not help here: wait time grows, operators start manually clearing the backlog, and manual confirmations eat up the remaining throughput.

Then accounting diverges. A debit in the database is not yet confirmed by the network, a stuck transaction holds a place in the queue and blocks the following ones — the address cannot sign a new operation with the same nonce.

Such operations are sped up by fee replacement or removed manually, and during this time withdrawals stand still, while balances have to be reconciled separately from network state.

Peak withdrawal load exposes the main thing: limits, retries, and double-send protection are almost always tuned for average flow, not worst case.

We break down how this is closed — by separating signing circuits, a fault-tolerant queue, and predictable degradation instead of a halt.

The effect of a managed signing circuit: speed, limits, failures

A managed signing circuit is verified not by a declaration, but by a set of measurements on the test and production circuits. Below is the checklist by which we accept the work: each item is either confirmed by a log and metric, or considered unresolved.

Keys are stored in a hardware module of the YubiHSM class — compromise of one node does not allow a signature to be assembled.

We test behavior during node failure separately: the backup circuit takes the flow, the withdrawal pause is measured in seconds, and the request queue is not lost.

We record measurements in the acceptance report so that limits and failure classes can be rechecked after every change to the scheme.

Hot circuit: signing and key storage modes

A hot wallet operates under load: withdrawal queue, automatic signing, limits per recipient address.

A key stored entirely in one process on the exchange server is a single point of failure: compromise of that process gives the attacker the entire reserve and nullifies the whole access separation scheme.

That is why the circuit is assembled so that no single node holds the key entirely, and the signing mode can be tightened without stopping request processing.

The schemes below differ in where key material lives and how many independent parties participate in assembling the signature.

Splitting signature shares with a threshold scheme (MPC) closes the single-point risk: the signature is assembled from shares, and the key never exists in one place.

In practice, a hybrid is more often chosen: splitting signature shares handles the fast flow, external storage confirms key provenance, and the isolated node covers large operations. The signing mode switches by volume thresholds, not manually.

Work stages: from flow audit to handover of procedures

We work on an exchange's hot wallet in a fixed sequence: first we understand how money actually moves in the system, and only then do we write code.

If you start with configuration, the key storage scheme gets locked in by accident — and it has to be reworked under load, when there are live funds in the accounts.

1. Flow audit. We look at how a withdrawal request passes from the external interface to signing: what queues, retries, where double-send is possible, who initiates the transaction and on whose command it goes to the network.
2. Key inventory. We record what keys exist, where they are stored, who has access, what duplicates and “temporary” solutions remain after quick launches.
3. Scheme design. We separate roles: a hot layer with daily limits for automatic withdrawals, warm and cold circuits for large amounts, a signing quorum.
4. Configuration and isolation. We assemble the circuits, close access on the least-privilege principle, enable logging of every signing operation, and configure rotation.
5. Load testing. We run peak scenarios: how the request queue behaves during mass withdrawals, what happens when the signing node fails.
6. Rollback plan. We describe and drill: transfer to cold mode, revocation of access, replacement of a compromised key, recovery from backups.
7. Handover of procedures. We provide documentation, the role matrix, instructions for the on-call shift, and train the team so the wallet is operated by the rules, not from memory.

Documentation, key schemes, and recovery procedures

Documentation for an exchange's hot wallet is not a report for the shelf, but a working circuit: the on-call shift uses it to make decisions when minutes count.

We hand it over as part of the result, not as an appendix to a release, and keep it in the client's repository so updates go together with the code, not as a separate document that inevitably becomes stale.

• Key scheme — where key material is generated, where it is stored, and which component accesses it; paths that must not exist in the hot circuit and points where signing goes to an external component are marked separately.
• Access matrix — which roles and through what confirmation mechanism can initiate signing, change limits, add addresses to the whitelist; for each role, the approval procedure and prohibition of unilateral action are specified.
• Rotation procedure — what events trigger key reissue (scheduled change, employee departure, suspected compromise), who starts the procedure, and how it is verified that new keys are accepted before old ones are revoked.
• Recovery instruction — step-by-step return of signing to the backup circuit, including integrity checks of backups and a test transaction signature check before returning to operating mode.
• Incident review procedure — what is logged automatically, which events page the on-call engineer, at what point payouts are stopped, and how the timeline is recorded for later review.
• Circuit documentation — deployment diagram, list of external dependencies, and responsibility boundaries between our team and the client's operations service.
• Change log — every edit to policies and limits with the reason, author, and date, so a decision can be reconstructed months later.

Such a set removes the main operational risk: when the circuit works, documentation is not needed, and when it stops, it is too late to reconstruct the logic from code. Describe the task — we will come back with an estimate and a plan.

Case study: reworking the signing circuit and withdrawal limits

An exchange with spot trading and a flow of small withdrawals kept signing in the same service as the request queue: keys lay next to the handler, and restarting the process risked re-signing. Below is what it was before the circuit rework, what was changed, and how the result was verified.

Before: signing in a shared process

Average signing time stayed around 40 seconds — the request waited for the signing process to become free, and at peaks the queue grew many times over. There was no idempotency key: after a restart, the transaction was signed a second time, and such cases were handled manually.

The share of automatic withdrawals did not exceed 38% — everything above the default limit went to an operator.

After: isolated circuit and three-level limits

Signing was moved to a separate circuit on a Ledger Enterprise hardware module: the key never leaves the device, and the withdrawal service only forms the request. Each transaction was assigned an identifier — a repeated request with the same identifier is not signed. Limits were split into three levels: recipient address, client, and the platform's daily turnover.

Signing time dropped from ~40 to 2-3 seconds, and the share of automatic withdrawals from 38% to 92%. Limit rejections and re-signings stopped occurring: over the observation period, not a single duplicate.

Post-implementation verification

For the first weeks, the new circuit ran in shadow mode — signatures were assembled in parallel with the old one, each verified by transaction hash and audit log entries.

After the switch, control latency measurements under load and regression for re-signing were run; there were no discrepancies with the reference.

How does a signing circuit differ from cold storage?

These are not two competing solutions, but two circuits with different tasks. The signing circuit serves the operational flow: user withdrawals, deposit consolidation, liquidity top-ups — everything that must execute within seconds and minutes.

Cold storage holds the bulk of assets and is connected to work only when the hot circuit has exhausted its limit.

The boundary between them is set by fund allocation, not convenience. The share in the hot circuit is calculated from the daily withdrawal peak: if the peak outflow fits within the reserve of keys and confirmations, the circuit handles it on its own and does not block the queue.

Withdrawal frequency matters more here than total volume — an exchange with rare large withdrawals and an exchange with a constant stream of small transactions require different reserves.

The withdrawal threshold is the key rule: a transaction above the threshold is not signed automatically, but goes to a scheme with multiple operator signatures or is executed from cold storage.

The threshold is tied to the maximum expected withdrawal and the risk profile, not the average value — otherwise the first atypical withdrawal hits a rejection and manual processing at the worst possible moment.

Funding the hot circuit is planned in advance: transfers in batches on a schedule or when a lower threshold is reached, rounded to amounts at which network fees do not eat the point of the operation.

A backup signing circuit is needed in case the primary is unavailable: a second set of keys and an independent withdrawal path, so that failure of one node does not paralyze payouts.

The decision is made by metrics: distribution of withdrawals by size, daily and hourly peak, share of requests to cold storage.

The practical outcome — speed within the limit is provided by the signing circuit, while everything above the threshold or outside the peak schedule goes to the cold reserve.

Doubts about keys, limits, and fault tolerance

Concerns about a hot wallet relate not to the signing scheme itself, but to how it will survive an incident and integrate into the existing circuit. Below are answers to the doubts that most often come up before work starts.

What happens if one key is compromised?

Keys are separated by purpose: some for hot withdrawals, some for internal transfers. Compromise of one does not open access to the others — daily and address limits cap the damage, and key rotation follows the procedure without stopping withdrawals. Transactions signed by a revoked key are filtered out by the log.

How does the circuit survive node failure, and what confirms signature correctness?

The signature is assembled by quorum: loss of one node does not block withdrawals, and the majority decides. The queue and retry counters are stored outside the node, so after a restart the circuit continues from the same transaction and does not duplicate debits.

Signature correctness is confirmed by a set of checks: address match, re-entry, rejection on an invalid key or exceeded limit.

Let's discuss the task: what to send for an estimate and plan

To come back with a work estimate and implementation plan, rather than general wording, we need inputs on your current scheme. Some points we can close in one conversation, the rest we clarify as we go — send them as they are, including diagrams and configurations.

• Target networks and assets: how many blockchains, native and tokenized assets, whether smart contracts, staking, and simultaneous payouts to many addresses are needed.
• Load inputs: peak withdrawal frequency, daily transaction volume, number of user addresses, acceptable latency from request to signature.
• Current key scheme: where keys are stored now, what is in the hot, warm, and cold circuits, whether a hardware security module is used, whether threshold signing or multisig is used.
• Signing initiation procedure: who forms the request, how large debits are approved, what withdrawal limits exist, and how operator rights are separated.
• Infrastructure: own servers or cloud, jurisdiction and isolation requirements, availability of a backup circuit and recovery procedure.
• Integrations: how internal services pass a withdrawal request, what has already been done for compliance checks, limits, and operation logging.
• What hurts now: manual signatures, delays, service failures, audit findings — specific cases help prioritize the stage plan.
• Launch timeline and regulator requirements, if already fixed.

Describe the task — we will come back with a work estimate, a key protection scheme, and a stage-by-stage implementation plan.