External Secrets Operator: why manual secrets break releases
A secret with database access lives in several places at once: in the description of cluster objects, in build pipeline variables, and in the settings of a service that is deployed separately.
An engineer changes the value in one place, the others remain with the old version — and the rollout fails on the very first request to the database. The code is built, checks are green, but the cause of the failure is not in the code.
Such a failure is expensive to diagnose: it surfaces as an authorization failure in application logs, and the team goes looking for the cause in the application, not in the mismatch of values.
Manual secret rotation adds a second layer — by the time you go around all consumers, some are already behind, some remember the old value. The more environments, the more often secret handling in the cluster diverges from what actually runs in production.
This is resolved by moving the source of truth: a single secret store from which External Secrets Operator pulls values and distributes them into cluster objects. Secret synchronization runs on a schedule, rotation becomes one operation in the store, not a walk through services.
The update reaches consumers without manual edits, and the release stops depending on who managed to fix what on their side.
We break down how this is covered: operator installation, binding to the store, access policies, and rollback procedures.
Measurable effect: rollout, recovery, failure class
We measure the effect before and after implementation on the same scenarios: release rollout, planned rollback, and recovery after a node failure.
Before automated secret delivery, release preparation was stuck on manual steps — export a value from the store, put it into the cluster, synchronize environments. For each release, that's dozens of operations, and each one is a point where a human will make a mistake.
The first measurable shift is release preparation time. When External Secrets Operator itself pulls values from the store and updates them in the cluster, the “fetch and place” step leaves the process, and the release starts from the same state as the previous one.
The share of manual operations in rollout drops by about an order of magnitude, and rollout stability no longer depends on who is on duty today.
The second is service recovery time. New workloads come up and request current values themselves, so key rotation and failure investigation do not require manual intervention.
In snapshots, this is visible as recovery shrinking from hours to minutes — within a single synchronization cycle.
The third and most illustrative is the failure class.
Incidents due to access desynchronization (different values in environments, forgotten rotation, a copy of a secret inside a manifest) move into a separate category: the system either converges to the store state or explicitly reports a mismatch.
By this share, it is easiest to prove that the change is not cosmetic, and the before-and-after metrics are taken using the same methodology.
Secret delivery modes: comparison of approaches and limitations
A secret reaches the application in one of three ways: as an environment variable, as a mounted file, or via a request to an external store at startup. The choice of path determines how often manual intervention will be needed and what happens at the moment of rotation.
| Delivery method | When it fits | Limitations |
|---|---|---|
| Secrets in git, encrypted | A single declarative flow, few environments | The decryption key is distributed separately, previous values remain in history |
| Built-in Secret objects of the cluster | One cluster, individual applications | Drift between environments, manual rotation, no single source of truth |
| Synchronization from an external store (External Secrets Operator) | Multiple clusters, frequent rotation | Read permissions required, dependency on store availability, synchronization delay |
| Mounting a secret as a file | The application re-reads the file without restart | Not suitable for all applications, environment variables are not updated this way |
Delivering secrets to the application by any of these methods does not cancel the cluster role model: access is still limited at the level of the namespace and service account.
A separate limitation is environment variables: they remain in the pod description and are visible in a process dump, so for sensitive values a mounted file is preferable.
There is no universal option: the more clusters and the more frequent the rotation, the more synchronization from an external store wins.
Next is how templates and update rules are configured so that the secret changes in the source and the application receives the new value without manual edits.
Implementation step by step: audit, design, setup, verification, handover
The work order is structured so that at each step a working configuration remains in the cluster: until the new scheme is verified, the previous secret delivery method continues to operate.
This way, migration does not coincide with an outage, and changes do not have to be rolled back blindly.
- Secret handling audit. We inventory what and from where gets into the cluster: values in manifests and repositories, environment variables, mounted files, store read permissions. The result is a map “workload → secret → source → owner”.
- Store access scheme. We choose a single source of truth, the cluster authentication model, and namespace-based separation so that the team sees only its own paths. We separately record how external rotation affects workloads.
- Cluster implementation. We install External Secrets Operator in a separate namespace, limit its permissions to the necessary minimum, and enable environments one by one — first staging, then production.
- Workload migration. We replace regular secrets with declarations and leave a period of parallel operation: the old and new methods coexist while there are no mismatches.
- Fault tolerance verification. We simulate store unavailability, an expired token, and value rotation: we check whether pods survive a restart and whether desynchronization alerts fire.
- Handover to operations. We hand over schemes, configurations, instructions for standard operations, and rotation procedures, and walk through the failure response order with the team.
What remains in the repository and cluster: configurations and policies
We hand over the kit as a repository plus exported schemes: for each artifact, it is clear who the owner is, how it is updated, and what will happen on failure. Below is what exactly remains with your team after implementing External Secrets Operator.
- ExternalSecret and SecretStore/ClusterSecretStore manifests — synchronization configuration: which keys are pulled from the store, how often the source is polled, which fields end up in the resulting Secret.
- Store integration scheme: the path to the external secret store, the authentication method via a service account, the mapping “key in the store → secret name in the cluster”.
- Access policies: what each service is allowed to read, namespace-based separation, prohibition of direct reading of values from charts and images.
- Rotation procedures: value lifetimes, who initiates replacement, how it is verified that a pod has re-read the updated secret without a manual restart.
- Failure handling rules: behavior when the store is unavailable, actions on mismatch with the source, which events go to monitoring and alerting.
- Secret documentation: a map of key owners, the procedure for adding a new value, instructions for the on-call engineer in case of an incident.
- Transition plan: the order of migrating already running secrets, rollback to the previous delivery scheme, criteria by which we consider the stage complete.
Case: removed manual secret rotation in a large cluster
Before implementation
The platform is about 180 services in Kubernetes; secrets for databases, queues, and external APIs were stored in manifests and build job environment variables.
Rotation remained manual: an engineer obtained new values from the store, went through namespaces, updated Secret objects, and restarted workloads.
The full fleet took up to three weeks, about 40 manual operations accumulated per month, and each one carried the risk of touching the wrong environment.
The cost manifested in two places. First, incidents: 5–7 failures per quarter due to expired keys and desynchronization between staging environments, each with a nighttime investigation.
Second, releases: rollout waited for a rotation window, and release time stretched from hours to 2–3 days.
What changed
We deployed External Secrets Operator and described an ExternalSecret for each workload, with HashiCorp Vault as the source. The operator itself pulls values and rebuilds the Secret when the store changes; automatic rotation runs on the store schedule, without human involvement.
We also configured behavior when the source is unavailable: the last valid value remains, the workload does not go down, and a synchronization delay signal is sent to monitoring.
What became
Manual secret operations dropped to zero, and rotating the entire fleet takes minutes instead of weeks. There were no secret-related incidents over two quarters, and release time shrank to hours: rollout is no longer tied to a rotation window.
Rollback is predictable — the previous version of the value is preserved in the store, and changing the reference in the manifest returns the service to it.
External Secrets vs built-in secrets: what's the difference
Built-in secrets are Secret objects that reside in the cluster’s etcd. The value is base64-encoded, access is delimited by roles, and the record itself lives in one place: in a manifest, Helm chart, or deployment script.
While there is one cluster and dozens of secrets, this is enough: a pod gets the value from an environment variable or a mounted file.
The limits of applicability begin where the secret is needed in several clusters or is changed by someone other than the one who issues the release.
A built-in object knows nothing about an external source of truth: an old manifest will overwrite the new value on the next apply, audit will be limited to the API access log, and no one controls the lifetime of the database password.
External secrets change the scheme: only the description remains in the cluster — which key to take and where to put it — while the value is pulled by the operator from a store such as HashiCorp Vault or a cloud secret manager.
Synchronization runs on a schedule or on an event, the result materializes into a regular Secret, so the application code is not rewritten.
The difference comes down to four points: a single source of truth instead of copies across clusters, rotation in the store instead of rebuilding manifests, a separate access log, and no secrets in Git or build artifacts.
The cost is an additional component in the cluster, operator read permissions on the store, and a new failure point under observation. The choice comes down to questions: how many clusters, who owns the secret, how often it changes, and whether audit is needed.
One cluster with infrequent rotation — built-in secrets are enough; a secret shared across environments or delivery from an external system — an operator is needed.
Technical concerns: fault tolerance, permissions, migration
External Secrets Operator is a controller that keeps cluster secrets synchronized with an external store: the value changes in the store, the object changes in Kubernetes.
Hence the main concerns of engineers — what happens when the store is unavailable, under what permissions the system itself runs, and how to migrate to it without downtime.
What happens if the store is unavailable?
Already synchronized secrets remain in the cluster and continue to work — the controller does not delete them. New values simply will not be pulled until communication is restored, the object will go into an error status, and we will set up an alert for this state.
Under what permissions does the system run?
A separate account in the store with read access only to the required paths. On the cluster side — a service account with a role model limited to specific namespaces: the controller cannot read secrets it is not bound to.
How does the phased migration work?
We start with one namespace and one key, compare the values with the current storage method, then migrate the rest. We leave the old objects as a rollback path until synchronization completes several rotation cycles without mismatches.
What happens during secret rotation?
The changed store value is overwritten into the cluster object, and pods receive it on restart or via an external signal. For applications that do not re-read configuration on the fly, we set a controlled restart so they do not pick up old credentials.
How do we recover after a failure?
The store registry versions values — the previous key is restored with one operation. The controller configuration and manifests are stored in a version control system, so the environment can be brought up from scratch reproducibly, without manual tweaks.
Let's discuss the task: what to send and how the work will proceed
To come back with an estimate and work plan, rather than with clarifying questions, it is enough to send a few inputs. The more complete the initial data on clusters, secrets, and access, the more accurate the work order and stage boundaries.
- Map of clusters and environments: how many there are, which environments they are in, what the current Kubernetes version is, and who manages them now.
- Secret stores: a cloud service, HashiCorp Vault, or a custom solution, and how secrets get into the cluster today — manually, via CI, or with scripts.
- Secret inventory: which ones exist, which services they are tied to, and which are critical enough to require security approval.
- Access model: service accounts, roles, network restrictions between the cluster and the store, and whether there is a separate environment for testing.
- Rotation and failure requirements: how often keys change and what should happen to the workload if the store is unavailable.
- Rollout order: which environments and services we migrate first, and which windows are acceptable for changes.
Then we review the inputs, agree on the secret synchronization scheme with your store, and provide an estimate with a plan: stages, scope of work, control points, acceptance criteria.
We conduct the implementation for your task — from a pilot environment to full migration, with reversibility at every step. Describe the task — we will come back with an estimate and a plan.