What Breaks in an HD Wallet with Multichain and Growing Load
The classic failure in an HD wallet looks mundane: an operator manually issues an address for a specific network, does not record the derivation path and index — and the deposit arrives at an address that is not in the accounting system. It can be recovered from the seed phrase, but only if the path used is known.
When dozens of issues happen per day and there are several networks, manual address issuance becomes the main source of discrepancies: some addresses remain outside monitoring, some are reused, and balance reconciliation diverges from actual.
Multichain adds a second layer: networks have different accounting models — sequential transaction numbers versus a set of unspent outputs — and different address formation rules.
One operator error, when an address is issued for the wrong network or with someone else's index, produces funds that physically exist on the blockchain but are not visible in the client interface. A separate story is credits in the wrong network, which the wallet cannot accept by design.
The third bottleneck is signing. While operations are signed by a single node with a sequential queue, at peak transactions wait their turn, some drop by timeout, the operator initiates a retry — and a duplicate or stuck operation appears.
Control is also lost here: there is no single log of who issued an address and when, no protection against re-signing the same operation, and no reconciliation between networks.
Each of the three failures is solvable separately, but together they mean one thing: the wallet is not ready for load and for several networks at the same time.
Below we break down how this is closed — from taking address issuance out of manual operations to a signing queue that does not depend on peaks.
Implementation Effect: Key Control and Speed of Operations
The first measurable effect is a unified key perimeter: generation, storage, and signing are brought together in one place, where the private part physically never leaves the secure zone. Only extended public keys go outside — that is enough to issue addresses and build balance reports.
Manual address issuance operations go away as a class: addresses are issued in batches along the required derivation path, in seconds, without support tickets or manual exports.
A batch of new addresses that previously took a full working day is generated with a single operator command.
Access recovery time stops being a separate project.
The BIP-39 mnemonic phrase and fixed BIP-44 paths produce a deterministic result: the same root expands to the same set of addresses on any compatible wallet, and the procedure comes down to a regulation that the operator follows from the instructions.
Roles are also separated: transaction signing is moved to an isolated perimeter with a threshold scheme, and the observer node sees only public data.
This closes an entire class of failures — key compromise on a hot server, loss of access when an administrator leaves, a single point of failure in one service.
Downtime losses also stop being a shared risk: payouts, accruals, and balance reconciliation run on different components and do not block one another.
The share of manual operations in the daily cycle drops to isolated confirmations — the rest is covered by signing rules and limits.
Options for Building an HD Wallet: Comparison of Approaches
Choosing an HD wallet implementation option means choosing who is responsible for the cryptographic core and how far your control extends.
The BIP-32 specification defines the key tree, BIP-39 defines the mnemonic phrase, and BIP-44 defines the path structure; each approach below implements them in its own way.
The difference lies in network support, freedom to change signing logic, and where the keys physically live.
| Implementation option | When it fits | Limitations of the approach |
|---|---|---|
| Ready-made key library and your own integration layer | one or two blockchains, launch within tight deadlines | network support is limited to what is already implemented; non-standard address formats are added manually |
| Custom build of the core and storage | around ten networks, your own signing rules, role separation | longer to first release; requires your own test vectors and cryptography expertise |
| Configuration of a platform wallet | standard deposit and withdrawal scenarios, standard assets | signing logic and key format are set by the vendor; not suitable if keys must live in your perimeter |
| Hardware security module for keys and a thin client | server-side keys, strict access regulation | not suitable for keys on a user device; higher requirements for redundancy and fault tolerance |
There is one practical criterion — the key storage model: where the key is born, where it is stored, who has the right to sign, and how recovery from the mnemonic works.
If the answers to these questions cannot be reduced to the settings of a ready-made product, platform options are eliminated, and a custom build remains — which is then compared by network support and scope of maintenance.
Work Order: Audit, Scheme, Setup, Verification, Handover
We carry out the work in stages with a readiness criterion at each one — the result is checked as we go, not only in the final demonstration. Below is the order in which HD wallet implementation proceeds in the customer's infrastructure.
- Infrastructure audit. We look at where keys and secrets are stored, what perimeters exist, and who can deploy. Done when bottlenecks and change boundaries are documented.
- Key hierarchy scheme. We design the derivation path tree, separation by networks and roles, and recovery policy. Done when the scheme is agreed in writing, before the first line of code.
- Module setup. We implement generation, signing, and recovery according to BIP-32/39/44 standards, and connect the required networks. Done when an address is generated and the signature is verified in each network.
- Test perimeter. We deploy a separate stand: recovery from the phrase, key rotation, rollback to the previous configuration. Done when the scenarios are reproducible in a clean environment.
- Security verification. We review access rights, try to bypass signing policies, and reconcile the operations log. Done when the findings are closed and a repeat check does not find them.
- Result acceptance. We demonstrate the work on the stand according to the agreed checklist and compare it with the stage criteria. Done when each checklist item is confirmed by the customer.
- Documentation handover. We deliver the scheme, regulations, and support instructions, and conduct team training. Done when the customer's engineers deploy the module on their own.
Access Policies, Schemes, and Documentation: What Stays with You
Development ends not with a demo stand, but with handover: your team keeps a set of artifacts that make it possible to operate, extend, and transfer the system to another contractor without reverse engineering.
Documentation lives next to the code, is versioned with it, and is updated with every policy change — otherwise, six months later, no one will be able to say why the limit is set exactly that way.
- Network and environment configurations: the list of connected networks, addresses of primary nodes and backup providers, fee, limit, and retry parameters — deployment is reproduced in your perimeter without us.
- Key hierarchy scheme: the derived key tree, derivation paths according to industry standards, mapping of branches to products and client portals, and a map of address correspondence and purpose.
- Access policies, roles, and rights: a role matrix (initiator, approver, treasurer, auditor, administrator), signing thresholds, daily limits, address whitelists, and delays for large transfers.
- Operator instructions: step-by-step procedures for issuing wallets, rotating keys, adding a new network, and releasing an address from quarantine — indicating who confirms what at each step.
- Key backup regulation: where and in what form shares and encrypted copies are stored, who has access, and how often recovery is tested in an isolated perimeter.
- Maintenance regulation: the procedure for responding to network updates, the composition of monitoring and alerts, request priorities and response times, and escalation points within your team.
A separate appendix describes integrations with external key storage systems at the Fireblocks level — recording which policies remain on our side and which move under your control.
At acceptance, we check not the completeness of folders, but reproducibility: your engineers use the instructions to bring up the environment, perform recovery, and change a limit without contacting us.
Case Study: A Unified Key Perimeter Instead of Manual Operations
The customer is a payment service with daily payouts to merchants across several networks.
Keys were created manually on a separate machine: issuing an address took 30–40 minutes, and each payout batch required about fifteen manual operations — from inserting the address into billing to signing the transaction.
Access recovery when the responsible person changed stretched to hours, and this was a real single point of failure.
We built a unified key perimeter.
The master seed is stored in an isolated vault without network access; from it, the account tree is derived via BIP-32, BIP-39 provides a backup phrase for emergency recovery, and BIP-44 separates derivation branches by network so that addresses do not overlap.
Signing is moved to a separate service: private keys never leave the perimeter, and only a ready transaction goes out.
What Changed in Numbers
Address issuance now takes seconds instead of a half-hour wait. A payout batch now requires one confirmation instead of fifteen manual actions, and access is restored from the backup phrase in minutes rather than hours of downtime. Only payouts above the limit, where a human decision is required, remain manual.
At Payout Hub, the result was summarized briefly: “Address issuance used to slow down the entire payout queue; now it is background work — the team focuses on the product, not on moving keys.”
How Custodial Key Storage Differs from Non-Custodial
The difference between the models is not where the key file physically resides, but who holds its material. With custodial storage, keys are created and remain on the service side, and the client receives only an interface for signing operations.
With non-custodial storage, the key is derived on the user's device from the mnemonic phrase, and the service sees only public data.
This leads to different risk profiles. In the non-custodial model, access recovery depends only on the safety of the phrase — a key cannot be administratively reissued, and support cannot “unfreeze” access.
In the custodial model, the main question is how key separation is arranged: a single node with the full key is a single point of failure, so the key is split into shares and assembled for signing under a threshold signature scheme, while the shares themselves are stored in secure key storage modules.
| What we compare | Custodial | Non-custodial |
|---|---|---|
| Key owner | service; the client has access to the signing interface | user; the key never leaves the device |
| Access recovery | according to the service's regulation | only by mnemonic phrase |
| Main risk | internal actor, node compromise | phrase loss, infected device |
| Key separation | shares on different nodes, threshold signature | one key; separation is rarely used |
A hybrid option — shares are stored on the service side, but the full key is never assembled on any single node — removes some of the risks of the custodial model while still leaving the client with manageable access recovery.
Common Technical Questions: Keys, Backups, Access
At the design stage, the same concerns come up: what happens to keys during a failure, who will restore access, and will there be a trace of operator actions. Below are brief answers for each point.
How are keys stored and backed up?
The BIP-39 seed phrase never leaves the secure perimeter and is duplicated on offline media in different locations. Addresses are derived deterministically via BIP-44, so one copy is enough to restore all accounts and branches.
What should be done if a media carrier is lost?
The copies are not linked to each other, so losing one carrier does not stop operations. Recovery is performed from the second copy, and address and balance matching is verified against the network before withdrawals are enabled.
Is the author of each operation visible?
Every operator action is written to the log: initiator, time, object, result. The records are linked by hashes, they cannot be altered after the fact, and the export goes to your monitoring system.
How does the wallet integrate with accounting?
The software interface covers customer address creation, transaction statuses, posting exports, and balance reconciliation. Internal identifiers are matched with accounts, so manual data transfer is not needed.
What does multisignature provide?
A 2-of-3 scheme separates the right of disposal: one key is in cold storage, and the others are held by responsible roles. A large withdrawal cannot proceed without an agreed signature, and compromising one machine does not open access to funds.
Let's Discuss the Task: What Is Needed for an Estimate and Work Plan
To come back with an estimate and work plan rather than rough timelines, we need to understand your current infrastructure.
Below is what we ask for in the first conversation: this is usually enough to estimate the scope, name implementation timelines, and propose a starting procedure without stopping operations.
- The list of networks and assets that the HD wallet must support at launch, and what is planned to be added later — the scope of work for each network directly depends on this.
- How keys are currently stored and issued: where the master seed is located, who has access to it, and whether a separate signing service is used.
- Signing scenarios and roles: who initiates a transfer, how many confirmations are required, and whether rights need to be separated between employees.
- Withdrawal requirements: limits, delays, rules for blocking suspicious operations, and the manual confirmation procedure.
- Integration scheme — which of your systems must receive balances, history, and addresses, and through which channel.
- How address derivation to a new level of the key tree should work, and what address reserve is needed without contacting the server.
- Backup and access recovery requirements: where seed copies are stored, and who and how restores access if a carrier is lost.
- Constraints on implementation timelines and permissible windows when production services can be affected.
This is enough to come back with an estimate and work plan: a breakdown by stages, delivery scope, and commissioning procedure. If some data is restricted, we will start with what is available: discuss the task, document assumptions, and clarify the rest as we go.