cert-manager in Kubernetes: why TLS certificates expire
TLS certificate renewal in a cluster is almost always configured — and almost always breaks unnoticed.
The certificate for incoming traffic has a limited lifespan, renewal runs as a background process, and that is exactly why no one monitors it: as long as the chain is valid, metrics look healthy and there are no errors in the logs.
Then the deadline passes silently. Applications keep running, deploys are fine, but browsers and mobile clients refuse the connection — and this is no longer a failure of one service, but a failure of incoming traffic across the entire domain.
Clients with strict trust chain validation fail with a TLS error, and manual certificate issuance at that moment stretches into hours: you need to issue a new one, put it into a secret, and restart the pods.
There are usually three causes for this scenario.
Renewal is configured, but no one investigates domain ownership verification errors: the DNS-01 record remains the same, the HTTP-01 path is closed by a routing rule or protection against external requests — and the controller silently retries until the very deadline.
Manual certificate issuance leaves secrets with the same names and references to the old object, so the renewed chain simply does not get into the pod. And the notification that the term is expiring is not tied to any alerting channel.
The cost of these little things is the same — service unavailability at the worst possible moment. We explain how to close this gap and what must be configured before the deadline passes.
What automatic certificate management in a cluster provides
In a cluster, certificates stop being a manual task: each one is described declaratively — a Certificate object references a ClusterIssuer, and the result goes into a Secret that the ingress reads.
A single point of management means that issuance, storage, and rotation are defined by one manifest, rather than by visiting panels of different services. The reduction in manual operations is measured in the number of steps: a new domain is a commit, not a ticket in a queue.
Automatic certificate renewal is tied to the remaining lifespan: the controller sees that less than a specified share remains until expiration and reissues the certificate in advance, updating the Secret.
The ingress controller picks up the new content without restarting pods, so the reduction in downtime is not "almost zero" but exactly zero: the renewal window is so wide that a connection break does not occur.
The main effect is an eliminated class of failures. An expired certificate forgotten on a test bench, a wildcard renewed in only one environment, a desynchronization between clusters — all of this stops being an incident and becomes a controller task.
Predictable behavior during mass renewal is also ensured: requests to the certificate authority go with rate limiting and retries, so a wave of reissues does not turn into an outage.
Monitoring remains: metrics on validity period, events on failed domain verification, an alert before the critical date arrives.
This set is the maturity checklist — renewal without a human, a single point of management, zero manual operations, and the absence of a whole class of incidents.
cert-manager operating modes in Kubernetes: comparison of approaches
<cite>cert-manager</cite> does not issue certificates itself — it describes the signing source through Issuer or ClusterIssuer objects, while Certificate defines domains and lifespan.
Automatic rotation, client-side trust, and how revocation works when a key is compromised depend on the choice of source.
The modes can be freely combined in one cluster: public certificates for external services, an internal CA for traffic between services, self-signed ones only on temporary contours. Below are four options and the limitations of each.
| Issuance option | When it fits | Limitations |
|---|---|---|
| Public certificate authority | External domains and clients that do not have a corporate root certificate | Domain ownership verification is required, there are limits on the number of issuances, and the certificate ends up in public logs |
| Internal certificate authority of the organization | Services inside the network, mutual authentication between services | Clients need to trust the root certificate; an issuance and revocation policy is required |
| Corporate secret store | There is already a centralized key store and access audit requirements | Synchronization between cluster and store state is required, and rights are granted separately |
| Self-signed certificates | Test and temporary contours, debugging | Browsers and external clients do not trust them; such a certificate is not transferred to production |
In practice, two or three modes usually coexist in one cluster: an external contour on a public CA, an internal one on its own, and a separate self-signed one for checks.
We fix the choice in the issuer scheme and rotation policy so that a new service does not pull a certificate manually and revocation takes minutes, not hours.
Stages of implementing automatic certificate renewal
Automatic renewal breaks not on certificate issuance, but at the edges: a domain that was forgotten to be added to the description, or a private key in a single copy.
Therefore, the implementation stages are built from inventory to acceptance — we verify the result of each step before it enters the working contour. 1. Audit of cluster configuration.
We collect all points where the secure connection terminates, record domains, validity periods, and key storage locations. The output is a registry with an owner for each entry. 2. Trust scheme.
We determine the certificate authority, the procedure for issuing intermediate certificates, and the method of domain ownership verification. Separately, we decide where keys are stored and how they are rotated. 3. Issuance setup.
We describe certificates declaratively, bind them to domains, and set a renewal window with a margin — 30 days before expiration leaves time to troubleshoot a failure without rushing. 4. Testing renewal on a copy of the contour.
We force-run the cycle in a test environment: we look at the chain, the order of certificate replacement, and the behavior of services when the issuing authority is unavailable. 5. Acceptance.
We set up monitoring of the validity period and early notifications, and run the full cycle on the production contour in an agreed window. 6. Handover to operations. We provide the renewal policy, key placement scheme, and a list of those responsible for responding to alerts.
After that, renewal proceeds without manual actions, and a failure becomes visible before the certificate expires.
What the team is left with after automation implementation
Automation without documentation creates dependence on whoever set it up: an engineer leaves — and certificate issuance turns into an unexplainable black box.
Therefore, the handover includes not only a working contour but also a set of artifacts that the on-call shift and new specialists can use without us.
- Interaction scheme — how a domain request turns into a ready certificate: the ownership verification method, key storage location, and the link between issuance and routing objects. A diagram plus a textual description of each step.
- Certificate documentation — an inventory of all issued objects: domains, types (single, wildcard, multi-name), validity period, domain owner, and where the certificate is used.
- Renewal policy — how many days before expiration renewal starts, what happens on a failed attempt, who receives the notification, and the order in which a certificate is issued manually.
- Access policies — roles and rights: who changes issuance settings, who reads secrets, who has access to manage DNS records for ownership verification.
- Instructions for the on-call shift — troubleshooting of typical failures: expired certificate, domain verification error, exhausted quota at an external provider — with an indication of what to check first.
- Monitoring and alerts — a dashboard for certificate lifespans and alerts at a threshold where there is still time for manual intervention.
- Rollback procedure — how to return the previous issuance configuration and how to temporarily disable automation without bringing down routing.
The kit is assembled for your contour: object names, panel addresses, and areas of responsibility are yours, not examples from documentation.
Case study: how we removed manual certificate renewal
Context: a distributed environment — four contours (development, test, pre-production, production) and three regions, about 60 domains and subdomains, including wildcard certificates for internal services.
Certificates were issued manually: an engineer kept a table of deadlines, submitted requests to the certificate authority, and distributed files across load balancers and controllers.
About 40 operations accumulated per quarter, and the expiration of one certificate meant an outage of the external interface and errors for clients.
cert-manager was installed as an operator in the cluster and issuance was described declaratively: a separate source for each environment, a shared one for top-level domains.
Renewal starts 30 days before expiration, the key is updated in the same namespace and is picked up without restarting the application. We separately exposed a metric for remaining validity: a warning at 21 days, a critical alert at 7.
What changed in numbers
- Scheduled manual operations: about 40 per quarter → 0.
- Incidents due to expired certificates: 9 per year → 0.
- New domain issuance: 2-3 days with approvals → 15-25 minutes from manifest edit to ready certificate.
- Renewal errors (domain ownership verification unavailable, zones diverged) used to surface at the moment of expiration — now they are visible via an alert three weeks in advance; the share of successful auto-renewals over the year is 100%.
Only DNS provider changes and one-off exceptions remained manual — they are described in the on-call instructions, and troubleshooting takes minutes, not hours.
What to do if certificates are still issued manually?
Migration to automation is not a switch, but a sequence.
At the first stage, two schemes coexist: cert-manager in the cluster does not touch already created Secrets, because the link between the Certificate description and the store arises only by the name and namespace that you set yourself.
As long as there is no description, manual certificates continue to work and are renewed in the previous order.
The first step is testing in a non-production namespace.
We issue a certificate for a test domain in the same way that production ones will later be issued: whether domain ownership verification passes, whether the full chain gets into the Secret, whether the ingress controller picks it up without restarting applications.
This checks not only issuance but also that the trust chain is complete — otherwise clients will see a warning that cannot be caught inside the cluster.
When the result matches expectations, the descriptions are transferred to real domains one by one: first the less loaded contour, then the rest.
Manual certificates are not deleted — they remain a working rollback until the automatically issued one has lived through a full renewal cycle.
Renewal is the main reason for all this. Automation updates the Secret in advance, before expiration, and the application reads the new file without manual intervention.
If renewal fails due to the certificate authority being unavailable or changed rights, returning to manual issuance takes one action.
Access is closed separately: the right to update Secrets and description objects, access to the domain zone for ownership verification, and headroom for the number of requests to the certificate authority.
Without this, automation works until the first DNS change or rights expansion — and turns manual again.
Technical doubts worth resolving before starting
Before starting, four questions usually come up: where to issue certificates, what to do with services outside the cluster, how domain ownership verification will proceed, and whether renewal will work without manual intervention. Below are short answers that resolve most doubts before work begins.
External or internal certificate authority?
We use an external public CA for everything that users and external clients see — browsers trust it without additional configuration.
For internal services, we deploy our own certificate authority: the root certificate goes into trusted stores of nodes and client machines, and issuance happens inside the contour without external dependencies.
A mixed scheme is normal: different issuing objects for external and internal traffic, so that a failure of the external CA does not affect service connections.
What to do with services outside the cluster?
Certificates are issued for them as well: the secret with the key and chain is read by an external load balancer or reverse proxy, or it pulls the update from a shared store.
What is critical is something else — that after renewal the secret is re-read by the process and does not remain in memory until a manual restart.
We test this on a bench: shorten the lifespan, wait for automatic renewal, and see whether the external consumer picked up the new certificate.
How will domain ownership verification proceed?
Via a web server — if the domain points to the cluster and there is somewhere to put the verification file.
Via DNS — if certificates are needed for a group of subdomains at once or there is no public entry at all; then the issuing object needs rights to change records with the DNS provider, and we grant them narrowly, only for the required zone. For internal domains, ownership verification is not required.
How to make sure renewal happens in advance?
Renewal starts noticeably before expiration, so a failure has time to get into monitoring rather than into an incident report.
We set alerts for certificate age, issuance errors, and exhaustion of ownership verification attempts, plus a control run on a test contour with a shortened lifespan.
After that, the only question is access to DNS and a backup verification channel — if the primary provider is unavailable.
Let's discuss your task and come back with a plan
So that the work estimate is based on facts and not assumptions, we need a snapshot of your infrastructure: how certificates are issued now, where keys live, and what exactly should be covered by automation.
Below is the minimum set of data from which a substantive implementation plan is assembled. If something is not at hand, we will sort it out on a call and clarify in correspondence.
- Composition of contours. How many Kubernetes clusters, where they are located (cloud, own servers, hybrid), how many namespaces and traffic entry points must be serviced. This determines whether one shared issuing authority is needed or separate ones for each contour.
- Current issuance scheme. Who issues certificates today — manually, with scripts, or via an external service; where private keys are stored; which types are needed: single domains, wildcard, internal names.
- List of perimeter services. Domains and subdomains for automation, marked as to which are public and which are accessible only from the internal network. This determines the domain ownership verification method and the scope of checks after switching.
- Accesses and integration points. How DNS records are entered, who manages zones, whether there is an external secret store, and how workloads connect to it.
- Timeline constraints. Hard dates, maintenance windows, whether a short TLS interruption during switching is acceptable, and audit requirements for key lifespan.
- Past experience. What has already been tried, what incidents with certificate expiration occurred, and what exactly was unsatisfactory in the previous solution.
With this set, we fix the scope of work, the order of stages, rollback points, and acceptance criteria. Describe the task in a convenient format — we will come back with an estimate and a plan.