PKI in Your Own Environment: An Expired Certificate Takes Down a Service
PKI inside a company almost always grows organically: first one certificate for a test environment, then dozens — for load balancers, message brokers, internal APIs, and partner integrations.
They are issued manually: an engineer logs in to the certificate authority, generates a key, copies the file to the server, and records the certificate expiration date in a spreadsheet that no one opens afterward.
While there are five certificates, the scheme works.
When there are hundreds and they are scattered across environments, scheduled expiration turns into an outage: an expired certificate takes down a service at night or during peak load, the symptom looks like a connection error, and finding the cause takes hours.
Service downtime causes more damage than the replacement itself — partner integrations stop, the queue accumulates unprocessed tasks, and the on-call engineer manually renews something no one remembered.
Manual certificate issuance does not scale for engineering reasons either: there is no inventory showing which key belongs to which service, there are no unified policies for algorithms and validity periods, and revoking a compromised key stretches over days. Each replacement is a separate operation with the risk of affecting a neighboring service.
Next, we break down how this is solved: centralized issuance and rotation via Vault, short certificate lifetimes, automatic reissue on schedule, and monitoring of days remaining until expiration — so a certificate-related outage stops being an event that requires a person.
What Managed Issuance and Key Rotation Provide
The main measurable effect of PKI is certificate issuance time. Instead of days of approvals and manual signing, a service gets a key-certificate pair in minutes: the engineer specifies the purpose and lifetime, and the certificate authority does the rest. The private key never leaves the machine and is not sent in chat.
Automatic certificate issuance fits into deployment: a new server, when brought into operation, obtains a certificate for its role itself and does not wait for someone to remember the request. A short lifetime becomes the norm because renewal stops being a manual operation.
Key rotation follows a schedule: the certificate is replaced before expiration, intermediate certificate authorities are updated along the chain, and the window during which a compromised key remains dangerous shrinks to hours. For old services that cannot renew themselves, a controlled transition mode remains.
Revocation is a single point. Employee termination, key leakage, or decommissioning a node is handled by one operation in the certificate authority: a revoked certificate stops being accepted by all services, not only the one where the configuration was updated in time.
The access audit log records who received a certificate, when, with which role, and for what period — it is used to investigate an incident without reconstructing the timeline from chat.
Together, this removes the "forgot to renew" class of failures: an expired certificate no longer takes down the production environment on weekends.
PKI Architectures: A Single CA or a Hierarchy with Intermediates
Trust in PKI rests on who signs certificates and where the key that created that trust is stored. A single certificate authority is the minimal scheme: fast onboarding, simple procedures, minimal entities.
The downside is a single point of failure: compromise of the key invalidates the entire chain and forces reissuance of all certificates at once.
A hierarchy separates areas of responsibility: the root certificate authority goes offline and does not participate in routine issuance, while intermediate CAs do the work — they can be revoked individually without touching the root.
Root key storage is designed separately: a hardware module or offline medium, access split among several holders, and procedures for commissioning and recovery.
| Option | When it fits | Limitations |
|---|---|---|
| A single root CA | Internal environment, a small fleet of services, and a short issuance cycle | Single point of failure: key compromise invalidates trust and requires reissuance of all certificates |
| Root CA + intermediates | Multiple teams and environments, external partners, need to delimit access | More expensive to operate: procedures, lifetimes, scheduled revocation of intermediates, chain control |
| External public certificate authority | Public services and domains opened by the client's browser | Dependence on external issuance rules and timelines; keys on the provider's side |
A hybrid scheme is more common than the extremes: internal services are served by their own root, public domains by an external public certificate authority.
In Vault, this is separated by distinct mount points for the root, intermediates, and issuance roles — policies and certificate lifetimes do not overlap.
Implementation Stages: From Inventory to Handover
PKI implementation is not "deploy a secrets store and start issuing certificates" but a sequence where each next step relies on data from the previous one.
If you skip inventory or fail to document the trust model, issuance will work, but on the first rotation or service owner change, forgotten manual edits will surface.
Below is the order in which we run the work so that the output is managed issuance and revocation, not a set of disconnected operations.
- Service inventory. We collect who connects where over secure channels: services, load balancers, message brokers, internal APIs, integrations with external systems. Separately, we record how certificates are renewed today — manually, by a scheduled script, or "when it goes down."
- Trust model. We determine the depth of the chain: own root and intermediate CAs, where their keys are stored, what issuance lifetimes and revocation policies we apply to different classes of services.
- Deployment and configuration of the secrets store. We bring up a fault-tolerant configuration, enable the PKI module, unpack the root and intermediate CA, and enable logging of all issuance and revocation operations.
- Roles and access. We delimit rights: who issues, who revokes, who reads private keys. Applications get access under their own credentials, not a shared administrative one.
- Issuance setup. We connect automatic renewal: an agent on the node obtains a new certificate before expiration, and the service rereads it without restart or downtime.
- Testing in the staging environment. We run scenarios for issuance, rotation, revocation, node failure, and loss of connection to the store. We measure recovery time and make sure rollback to the previous certificate is possible.
- Handover to operations. We hand over configurations, trust models, revocation procedures, and instructions for the on-call shift, and go through typical failures with the customer's team — after that, the system lives on without our involvement.
PKI Documentation, Diagrams, and Procedures
The work ends not at launch, but at the moment when your team issues a certificate, revokes a compromised one, and closes an incident without calling the contractor.
Therefore, along with working infrastructure, we hand over a set of artifacts: it describes not "how we did this" but how to maintain it further.
- Trust model — the tree of certificate authorities, lifetimes of root and intermediate certificates, storage locations for private keys, and the circle of people with access to them.
- Access policies — roles and rights for issuance, renewal, and revocation, separation of duties: one engineer cannot both issue a certificate and confirm its revocation.
- Rotation and revocation procedures — reissuance timelines by certificate class, what is automatic, what is manual, the maintenance window, and the procedure for key compromise.
- Configurations and code — store settings, role and policy descriptions, issuance scripts in version control: the environment comes up from scratch without manual operations.
- Integration map — which services depend on PKI and how their reissuance happens without downtime: scheduled replacement separately, emergency replacement separately.
- Instructions for the on-call shift — scenarios for typical failures: an expired certificate on a load balancer, the store unavailable, suspected key leakage, with escalation points.
- Monitoring and checks — notifications about approaching expiration, verification of the trust chain and rights, and a PKI status report for the on-call engineer.
Handover happens through a walkthrough in the staging environment: your engineers issue and revoke certificates themselves under our supervision, rather than reading instructions after the team leaves.
Case Study: Eliminated Downtime Due to Expired Certificates
Before switching to automated PKI, about 40 internal services received certificates manually: a request in a queue, an engineer issued a file and distributed it across nodes, and renewal lived in one person's calendar.
Per quarter, there were 8–10 outages due to expired certificates, and recovery took from 20 minutes to several hours — first they looked for the culprit, then reissued and restarted.
A side effect: deployments stopped being scheduled at the end of the week so there would be time to react.
What We Changed
We deployed an internal certificate authority on HashiCorp Vault: the root key is isolated, intermediates are issued by roles for a specific service, and the lifetime ranges from hours to several days.
Issuance and renewal moved to automation: the application requests a certificate itself, the update is picked up without restart, and revocation of a compromised certificate happens via the revocation list within a minute.
What Changed
- outages due to expired certificates: 8–10 per quarter → none in a comparable period;
- certificate issuance: hours of waiting → seconds, without requests and approvals;
- recovery after a failure: hours → minutes, with rollback to the previous intermediate certificate without stopping services;
- share of services with auto-renewal: 0 → 100%.
Fault tolerance stopped depending on one engineer: the store is spread across zones, unseal keys are split among responsible people, and reissuance happens on schedule — a short certificate lifetime is no longer a risk but the norm.
How Internal PKI Differs from a Public CA
The difference is not in cryptography — certificate formats and algorithms are the same.
The difference is in whose trust store already contains the root certificate of the CA: a public certificate authority is preconfigured in browsers, operating systems, and mobile platforms, so its certificate is accepted without configuration.
The root of an internal PKI becomes trusted only where it was installed by you — on managed servers, in clusters, and on employees' work machines.
This defines the boundary of use. A public CA is needed wherever a certificate will be seen by someone else's device: a customer-facing website, an open API, an app from an app store, a payment gateway, exchange with partners.
Internal PKI covers services that an external user cannot reach: mutual service authentication, traffic encryption between databases and proxies, administrator access, and tunnels into office networks.
Trust in a certificate is not a binary attribute but a question of who can add the root to the trust store. As long as the fleet of machines is managed, distributing the root is a routine operation. On a client's or partner's device, there is no such lever, and an internal CA is useless there.
The choice of scheme is therefore more often hybrid: a public CA serves the external environment, and an internal one serves the internal/service environment.
In this combination, Vault takes on issuance and rotation of short-lived certificates at the service's request: the intermediate CA is configured once, and renewal then happens without manual requests.
The internal environment gets a certificate in seconds, and the external environment remains compatible with any browser.
Technical Concerns: Revocation, Rotation, Fault Tolerance
Five questions that most often come up before the start: the fate of already issued certificates, key compromise, issuance without manual requests, fault tolerance of the store itself, and access to root keys.
What happens to already issued certificates upon revocation?
A certificate does not disappear instantly: it goes into a published revocation list, and services that check status stop accepting it. A short lifetime reduces the window — we issue service certificates for days, not years.
A key is compromised — how quickly will you revoke it?
The private key is generated inside the store and never leaves it, so file leakage does not happen. If compromise is suspected, we revoke it with an automated scenario in minutes and issue a new one — the service does not need to be stopped.
Can certificates be issued without manual requests?
Yes. A service requests a certificate under its role with domain restrictions, and issuance takes seconds. Manual confirmation remains only for root keys — there the risk of an error is higher.
What happens if the store is unavailable?
Already issued certificates continue to work — they are validated on the service side. Issuance of new ones survives an outage: nodes are combined into a cluster with automatic failover, and the standby node accepts requests.
Who gets access to root keys, and how are backups stored?
The root key is assembled using secret sharing — parts from several responsible people are required, and it never exists in full anywhere. Backups are encrypted and stored outside the main store, and recovery is tested in drills.
Let's Discuss the Task and Come Back with a Work Plan
To make the estimate specific rather than a market average, we need brief introductory context: which services issue and accept certificates, how many there are and how often they are renewed, whether there is an in-house certificate authority, and who is responsible for root keys.
An hour of conversation with your engineer and an environment diagram is usually enough to understand whether we are building PKI from scratch or starting from the current state.
Configuration review and the return with an estimate and work plan are done by the same engineer who will later configure the environment — not the sales department. At this stage, our infrastructure and cryptography specialists get involved.
- Scope of work — certificate issuance and revocation, roles and access policies, automatic rotation, root key backup, expiration monitoring.
- What we need from you — access to a test segment, a list of consuming services, and a decision-maker for policies.
- Start procedure — first the environment boundaries and trust model, then configuration in the test segment, then migration to production without stopping services.
- Checkpoints — after each stage we show the result: working issuance, revocation, and chain verification.
- Handover of the environment — procedures, recovery instructions, and a walkthrough with your engineers on the live environment.
- Scope boundaries — we fix them before the start so there are no ambiguities regarding scope and timelines.
Describe the task — we will come back with an estimate, work plan, and start procedure.