Why HSM setup stalls during integration
Signing keys on blockchain projects almost always start the same way: they are generated by a script, put into a file next to the application, and handed to the service that issues transactions. While only one service signs, the scheme looks workable.
As soon as wallets, the node, the bridge, and internal payouts start signing, each has its own copy of the key, its own file with a password, and its own version of the “correct” environment.
Then security breaks. Storing keys in the file system means that access to the private key equals access to the server: anyone who reads the build directory or gets into the container takes away the ability to sign.
An engineer changed, a contractor left, a backup copy leaked — the key has to be rotated along with address migration and stuck transactions.
Release suffers no less. Manual key operations — running a script, entering a passphrase, manual confirmation — keep the release dependent on a person: a nightly rollout waits for the on-call engineer, and signing a transaction batch turns into a separate coordinated procedure.
HSM turns the scheme around: the key is generated inside the module and physically never leaves it, the application sends only data for signing, the result is returned outside, and protection of the key infrastructure no longer rests on team discipline and is confirmed by an access log.
It is precisely in the transition to this model that HSM setup stalls: signing code has to be separated from key material, roles have to be split, calls have to be rewritten, and the rotation policy has to be agreed. We’ll break down how this is addressed.
Implementation effect: fault tolerance and key control
While private keys sit in files on the host or in environment variables next to the signing service, we have one failure class with the maximum cost: access to the server means a copy of the key, transaction signing on behalf of the service, and a complete change of key infrastructure.
The second class is operational: rotation, unlocking after a restart, and backup reconciliation are done manually, and each such operation opens a window for error.
A hardware security module removes both. The key is generated inside the device and is not exported through either an administrative or cryptographic interface: the service sends data and receives a signature, while only a log entry goes inside.
Compliance with FIPS 140 means that key material is erased when someone attempts to open the enclosure.
Fault tolerance of signing rests on module redundancy: devices are combined into a group, the key is stored in each under quorum control, and failure of one does not stop the transaction flow.
Recovery is replacement of the device and assembly of the key copy using a secret-sharing scheme among holders. Downtime is reduced from hours of coordination to one cycle of scheduled maintenance.
Auditing key operations stops being a matter of collecting logs from different systems: the module keeps an immutable log of generations, rotations, and policy changes.
Key compromise moves from the “reissue everything” scenario to a controlled procedure with a precise time boundary.
HSM connection options: setup for your tasks
Choosing where to place the module is not a matter of taste: it determines who physically holds the key material, what latency appears on the signing path, and what happens if the site fails.
A local module, a leased environment from a provider, and a hybrid scheme cover different risk profiles and pass audits differently.
Comparison of approaches comes down to three questions: where the root key is stored, who has physical access to the security partition, and how access is restored after loss of the site. The answers are summarized in the table.
Module placement option
| Option | When it fits | Limitations |
|---|---|---|
| Local module in your own environment | Full control over keys is required; signing takes place at one site, and latency matters | Requires access-controlled premises, a separation-of-duties scheme, a replacement plan for failure, and an update policy |
| Cloud module at a provider | No own site, signing operations are rare, need a quick start | Keys are with a third party, dependence on the communication channel, some audit requirements cannot be met |
| Hybrid scheme | The root of trust remains local, operational keys are in a leased environment; a reserve is needed in case of site loss | Two access models in one policy, synchronous key rotation, separate control of separation of duties |
Switching between options does not require reissuing keys if the module supports a standard interface for accessing them. Therefore, the scheme can start with a leased environment and later move the root of trust locally — without stopping transaction signing.
Module implementation stages: from audit to handover
We fix the order of work before the start: rearranging stages after deployment means a repeated key ceremony and new shares for the responsible parties. Each stage is closed with an artifact accepted by the customer.
Order of work
- Key infrastructure audit. We inventory keys and signers: where they are stored now, who has access, which operations are critical, and what the auditor requires.
- Connection scheme design. Number of modules and sites, quorum access policy, operator roles, network isolation, and planned share rotation.
- Key ceremony. Keys are generated inside the module and split into shares; the private part does not leave the secure perimeter.
- Deployment and integration. We connect signing services and network nodes through a standard crypto provider, and describe behavior when the module is unavailable.
- Recovery verification. We bring up the configuration from backups on a separate test bench, simulate a failure, and measure the time to return to operation.
- Fault tolerance verification. We shift the load to the backup module, check operation logging, and verify that alerts to the on-call shift are triggered.
- Handover to operations. We hand over the scheme, access policies, ceremony policy, and instructions for the on-call shift, and conduct training.
Documentation package and key infrastructure diagrams
Upon completion of the work, the customer has not only a working module but also a complete set of artifacts by which their own team can manage keys without contacting us.
Each document describes the actual state of the installation — current policy parameters, real roles, and verified procedures, rather than a generic template.
- Key documentation: a registry with identifiers, purpose, binding to blockchain addresses, and protection level for each key, including a “key — service — owner” mapping.
- Access policies: a role matrix with signature quorum for critical operations, separation of duties between administrator and operator, and emergency access rules with mandatory audit log recording.
- Rotation policy: frequency of scheduled rotation and conditions for unscheduled rotation, procedure for decommissioning old key material, and mandatory signature verification on a test environment before switching the production one.
- Redundancy scheme: topology of the primary and backup modules, copy storage locations, recovery procedure after failure, and checkpoints at which integrity is confirmed.
- Operating instructions: typical failure signals, degradation criteria, actions for the on-call engineer, and boundaries beyond which intervention must be coordinated.
- Knowledge transfer to the team: walkthrough of the configuration on a live installation, practice of rotation and recovery scenarios on a training environment, and a separate session on non-standard situations.
Documents are stored in a version control system alongside the configuration: when policies change or a new service is connected, edits go into the same set, so the policies do not diverge from the actual state of the infrastructure.
Case study: how manual operations with signing keys were removed
Before: signing depended on the on-call engineer
The customer is a payment service provider with a round-the-clock transaction flow.
Signing keys were stored on an isolated host, and each batch was signed by the on-call engineer: they took out the storage medium, confirmed the operation according to the procedure, and put it back. One signature took 30–40 minutes, and the transaction queue waited all that time.
The manual operation produced two classes of failure. Human — a forgotten shift, a mixed-up storage medium, a mistake in the sequence of actions: signing did not go through, and the service went down for 2–4 hours.
Procedural — while the key was being extracted, it existed outside the secure perimeter. In a quarter, 3–4 incidents with downtime and investigation accumulated.
After: keys stay in the module, signing is an interface call
We deployed a hardware security module and moved signing to a software call. The key is generated inside the module and never leaves it — only the signature result goes out.
Access is divided by roles and a k-of-n scheme: one operator initiates, the second confirms; rotation runs on schedule, without stopping the service, and every call is recorded in the log with the initiator.
Metrics: batch signing takes seconds instead of dozens of minutes, manual operations are zero, and signing queues have disappeared.
Over the observed period, there was not a single downtime due to signing; the “forgotten shift” and “wrong storage medium” classes are excluded architecturally, not by procedure. Rollback to the previous key was tested on the installation — minutes.
How does a software wallet differ from a hardware module?
A software wallet is an encrypted container with a private key on the server disk or in a secrets store. The key is decrypted into process memory, and at the moment of signing it can be read by anyone who has access to that process, to a memory dump, or to a backup.
The difference between the approaches is exactly here: where the key physically resides and whether it leaves its storage even for a moment.
A hardware module (HSM) is built the opposite way: the key is generated inside the secure perimeter and never leaves it. Only the finished signature or the result of a cryptographic operation is returned outside.
Even full access to the server — with superuser rights, with a disk copy, with reading all RAM — does not allow extracting the private key.
Protection levels differ not by key length but by the trust model. For a software solution, it is process isolation, access separation, and external auditing of operations.
For a module, it is physical protection of the enclosure, key erasure on attempted opening, separation of operator roles, and quorum: a signature is assembled only after confirmation by several responsible parties, and every access goes into an immutable log.
When a module is needed: the key manages issuance and movement of client assets, treasury operations, a network node, or a wallet with multiple signers, and also if audit and regulatory requirements require proof that the key never left the secure environment.
For test environments, internal services, and tasks with a single operator, process isolation and encryption are usually sufficient — the transition is justified when the risk of key compromise outweighs the complexity of operation.
Technical doubts worth resolving before work starts
The doubts here concern not the module itself but its integration with an already operating environment: what to do with existing keys, whether a separate test installation is needed, and how to avoid ending up with a single person holding full access.
Compatibility with current services is resolved without rewriting logic — the module fits into existing signing and verification processes.
What to do with already issued keys?
They are not reissued blindly: migration is done in a protected form — in an encrypted envelope or through secret sharing among several holders, each entering their part separately.
If migration is impossible under policy, the key remains active with a narrowed scope, and a new one is issued alongside — with a period during which external systems can switch without changing addresses and identifiers.
Is a separate test environment needed?
Yes, and it is deployed before production, on its own key hierarchy and without access to production systems. The installation runs through ceremonies, the recovery procedure, and system behavior when one module fails — including the scenario where it does not come up at all. Drills that break the installation do not create risk in the production environment.
How should access be separated?
Roles are divided into at least three: the administrator configures the module but does not sign; the operator signs but does not change policies; the auditor only reads the operations log.
Critical actions — changing policies, extracting a key, changing the quorum — require simultaneous confirmation by several holders and are not performed by one person alone.
What if the module fails?
A backup module with an identical policy and separately stored key copies remove the single point of failure.
The procedure for entering secret shares and the integrity check are practiced in advance, so access recovery follows the policy rather than happening in a troubleshooting mode.
Let’s discuss the task and come back with an estimate and plan
To come back with an estimate and plan rather than clarifying questions, a short introduction is enough: how keys are currently stored, which services sign transactions, and who has access to them.
What to send for review:
- placement diagram: where network nodes are, where the module is, how segments are separated, and who initiates the connection;
- list of signing services: nodes, asset storage service, contract deployment, offline signatures;
- role model: who initiates, who confirms, whether a scheme with multiple administrators is needed;
- audit requirements: what goes into the log, who restores the key and on what basis;
- fault tolerance mode: one module or a group, acceptable signing downtime;
- rotation policy, load limits, and key lifetimes.
Then there are three steps. First, review: we compare the architecture with the requirements and show where the configuration will diverge from reality.
Then, an estimate of the scope of work — stage composition, number of iterations, decisions needed from you, and risks with ways to mitigate them.
After that, an implementation plan with the order of work, acceptance points, and rollback procedure, so that changes are introduced gradually and without stopping signing.
We will need access to the test environment, a person responsible for cryptography, and a window for the work.
We formulate the requirements for the contractor as follows: reproducible configuration, documented key policies, a verified recovery scenario, and knowledge transfer to your team.
Describe the task — we will come back with an estimate and implementation plan.