Secrets Management in GitOps: Where the Delivery Pipeline Breaks
Typical picture: next to the manifests in the repository there is a file where passwords, tokens, and keys are written in plain text. Deployment from this repository goes green, everything works — until the first audit, an engineer leaves, or the contractor changes.
The problem is not one file. Credentials sit in manifests in plain text: database connections, access to external services, encryption keys — all in the clear for anyone with access to the repository.
Rights to them are granted manually: an engineer opens the console, creates an account, grants a role, and records none of it anywhere. There is no rotation — a key lives for years, and no one can say how many copies have already spread across local machines and backups.
The commit history remembers everything: even if you delete the file and commit a cleaned-up version, the value remains in the history of secrets in commits — that is, in every clone of the repository for everyone who has ever made one.
A credential leak from the repository is not discovered immediately: logs show that the key is being used, but they do not show that it is already out of control.
This is most costly at handover: an engineer leaves or the contractor changes, and no one can list the live accesses, who they were granted to, and what will break when they are revoked.
Manual reconciliation of consoles against the repository remains — with untraceable rotation and no guarantee that old keys are already dead.
We break down how this is closed: how secrets travel through the same delivery pipeline as all other manifests, while no plaintext values remain either in the repository or in its history.
What Changes After Implementation: Speed, Access, Failures
The move to secrets management via Git is noticeable not by the number of tools, but by three changes: environments become reproducible, rotation stops being a manual operation, and configuration drift stops taking services down.
Environment Reproducibility
The pipeline is raised from the repository as a whole: manifests, access policies, and encrypted values sit side by side and are versioned. The test environment gets the same secrets structure as production; only the values differ, so behavior matches.
Service recovery time after an incident comes down to applying manifests, not searching for passwords in chats and notes.
Managed Rotation Without Stopping Services
Changing a key or password stops being a manual walk through services. The value is edited in an encrypted file or in an external store, the synchronization operator pulls the new version, and the application picks it up as pods are updated.
The old secret is revoked as a separate step — a window remains when both versions are valid, and the rollout proceeds without downtime.
Removed Failure Class
Configuration drift between environments is closed by having a single source of truth.
A change goes through review and lands in Git, not manually into the cluster, so there is no “the fix was made in prod and stage was forgotten.” If the actual state diverges from the described one, synchronization returns it to the repository, and the incident does not grow.
Comparison of Approaches: In-Repository Encryption and External Stores
Both approaches remove manual creation of secrets in the cluster, but they pay for it differently.
File encryption leaves the decryption key inside the cluster and requires re-encryption on every rotation; an external store adds a network dependency and a second fault-tolerance loop.
The choice is determined by the number of clusters, rotation frequency, and the requirement to prove who read a value and when.
| Option | When It Fits | Limitations |
|---|---|---|
| File encryption in the repository (SOPS, Sealed Secrets) | one or two clusters, a small team, dozens of secrets | the decryption key is still in the cluster; revoking access means re-encrypting all files |
| External store and synchronization operator (ESO) | multi-cluster, several teams, scheduled rotation | a fault-tolerant store loop and network availability are required; its own failure class |
| Hybrid: bootstrap secrets in the repository, runtime secrets external | access to the store and initial initialization | two sets of rights and two rotation procedures |
What You Pay for File Encryption in the Repository
The decryption key lives in the cluster, so separation of decryption rights is coarse: access to a namespace opens all secrets in it. Rotation means re-encrypting all files and a new commit, while previous ciphertext versions remain in the repository history. Diffs are unreadable: review shows a changed blob, not a changed value.
When an External Store Is Justified
An external store is justified when there are hundreds of secrets, more than two clusters, and rotation runs on a schedule: access revocation is instant, and the access log records reads.
The price is that network access becomes a critical path: if the operator does not cache values, store unavailability stops the rollout.
Implementation Stages: Audit, Design, Configuration, Verification, Handover
The order of work is based on the principle: first we record the actual state, then we design the pipeline, and only then do we change production environments. Secrets are migrated one at a time, so releases do not stop.
What the Current-State Audit Looks Like
Inventory takes one or two days and does not require access to the secret values themselves.
We gather where they currently reside — environment variables, files on servers, values in repositories, manual records in databases — who reads them, what happens during rotation, and who holds master keys.
The output is a dependency map and a list of places where a secret appears in plaintext: commit history, build logs, configuration dumps.
- Secrets handling audit — dependency map, leak points, critical services.
- Delivery pipeline design — what is encrypted in the repository, what lives in an external store, how the value reaches the pod.
- Access policy configuration — roles, namespaces, key scope.
- Migration of existing secrets — one at a time, with temporary parallel operation of the old and new schemes.
- Integration into the build and release pipeline — checks for plaintext values getting into a commit.
- Failure and recovery verification — restart, node loss, key revocation, release rollback.
- Handover of the pipeline to the team — accesses, rotation policy, walkthrough with on-call engineers.
How the Customer Accepts the Pipeline
Acceptance is running scenarios on a copy of the pipeline, not a demonstration: we add a secret, change a value, revoke access, take one node out of service, and see whether delivery recovers.
The customer repeats the key scenarios together with our engineers and receives the verification log and rotation policy — after that the pipeline is considered accepted.
Access Schemes, Rotation Policies, and Pipeline Documentation
We assemble the documentation package so that the on-call shift and a new engineer can understand the pipeline without us. The materials sit next to the manifests in the same repository and are changed through review, not by an email.
- Pipeline diagram: environments and trust boundaries, which secrets stores are connected to the repository and how they relate to each other.
- Access policies: read and write roles, environment separation, least-privilege principle, procedure for granting and revoking access.
- Rotation policy: table of secrets with replacement period, procedure for changing the encryption key, who starts rotation and by what signs confirms the result.
- Recovery instructions: what to do when the root key is lost or compromised — from revoking values to verifying that workloads received new ones.
- Environment documentation: list of namespaces, mapping of services to stores, purpose of variables, actions on drift.
- Responsibility matrix: who owns the pipeline, who is on call, whom to contact during an incident.
How These Materials Live After Handover
Documents are versioned together with manifests: changes to them go through the same review, so the instructions do not diverge from the actual state of the pipeline.
Once a quarter we reconcile the rotation policy with the change log — if a secret was replaced off schedule, the reason goes into the document.
We verify recovery separately on a training pipeline. If the steps do not add up, we fix the instruction rather than making verbal agreements. This way the package remains a working tool, not a folder opened once during acceptance.
Case Study: Removing Secrets from the Repository Without Service Downtime
A pipeline of three clusters and fifty services kept secrets directly in repositories: database tokens, external system keys, certificates. Anyone with access to the project could read them, and the edit history stored old values.
The migration was phased — first a pilot on non-critical services, then production — so as not to stop services.
What It Was Before
Rotation of one key stretched over two to three weeks: the change was prepared manually, approved, waited for a maintenance window, and services were restarted.
Rolling back a change meant restoring from a backup and reconciling values manually while part of the pipeline still ran with the old key. Measurements before: outside the repository — zero secrets, average rotation — 18 days, downtime during rollout — up to 40 minutes.
What It Became After
In-repository encryption was closed through Sealed Secrets: decryption is possible only in the cluster, and applications receive values by role.
Rotation in the production pipeline is one commit to a protected branch; the value spreads across all clusters in tens of minutes without stopping services.
Rollback is a normal commit revert; verification on the production pipeline runs on the pilot earlier than on production clusters.
“We tested the scheme on the production pipeline at every step: first the pilot, then production. The main thing is that rotation stopped being an event with downtime and a maintenance window,” — a team engineer.
How to Rotate Secrets in GitOps Without Stopping Services?
A secret does not get into a pod by itself: an agent injects it, and the application reads the variable at startup.
That means rotation is not replacing a value in the store, but a sequence: first we issue the second version, then switch consumers, and only then revoke the old one.
The key technique is an access overlap window. Two versions live in the store at the same time: the new one and the previous one, still valid.
The application receives the new one through External Secrets Operator, while the old one remains operational until the last instance switches to it.
If one pod lags or holds a connection to an external resource, it keeps working — zero-downtime rotation rests precisely on this overlap.
Next is liveness verification. Pods are replaced one by one, no more than one at a time: a readiness probe and a real request to the dependent resource confirm that the new access is actually accepted. Only when the whole set responds does the rotation window close and the old version is revoked.
If verification fails, an automatic rollback is triggered: we return the previous version of the secret, pods re-read it, the state stabilizes, and the engineer receives a notification with the reason. A failed rotation turns into one extra restart instead of downtime.
The practice is simple: rotate one service at a time, keep the overlap window short, choose a time outside peak load, and keep a log of issued versions. Rotating everything at once is the surest way to get the very downtime we are avoiding.
Engineering Doubts: Encryption, Access, Rollback, Audit
Technical objections from customer teams are almost always the same, so we answer with the order of actions and applicability boundaries, not promises. The scheme works where secrets are described declaratively and appear in the cluster from git or an external store.
Who decrypts the secret, and where?
It depends on the tool. Sealed Secrets reveals the value only to the controller inside the cluster — the private key does not get into the repository or onto engineers' workstations. SOPS stores ciphertext in git and reveals it with an age or GPG key at apply time.
External Secrets Operator does not keep values in the repository: git holds a reference to an entry in an external store, and the object is assembled in the cluster.
How do you separate rights between teams and environments?
Rights are split into two loops: access to git and access to decryption. In the repository, mandatory approval and CODEOWNERS by environment directories apply, while decryption keys are issued separately for each environment.
Then a developer can edit a test environment manifest but cannot reveal production pipeline secrets.
What happens when rolling back to a previous release?
Rollback is a commit revert: the previous version of the secret is returned by the same apply as any other change. The one condition is that the value must remain in the source.
The cluster controller or external store keeps version history; otherwise rollback returns a reference without data, and the pod with the secret will not come up.
How is access to secrets audited?
Audit is stitched together from three logs: commit history and change author in git, the cluster API access log, and the issuance log in the external store.
By secret name and time, you can see who changed the object, which controller read it, and whether there was access outside the normal pipeline.
Secrets that cannot be put in git even encrypted — root tokens, initial configuration, the store's own keys — are moved to a separate manual-entry loop and are not transferred to the repository.
Let’s Discuss the Task and Come Back with an Estimate and Work Plan
Secrets in GitOps break not at the stage of installing the controller, but at the joints: decryption inside the cluster, rotation, service account permissions, behavior during rollback to a previous revision.
To make the scope estimate substantive rather than “plus or minus,” we start with a breakdown of the current scheme — how it is arranged for you now.
What we review on the call:
- Review of the current scheme: where the source of truth for secrets is, how they get into the repository, what is decrypted and at which step.
- Failure points: storage of encryption keys, service account accesses, drift between environments, rollback scenario.
- Comparison of options — Sealed Secrets, SOPS, external secrets operator — with the limitations of each as applied to your scheme, not in general.
- Order of steps: what we migrate first, where we leave a manual loop for the transition period, how we verify that the secret arrived.
- Scope estimate: composition of changes in manifests and roles, rotation policy, responsibility area for keys.
- What remains with the team: documentation, secret templates, checks in the delivery pipeline before changes are applied.
Such a review takes one conversation and immediately shows where you are already covered and where manual work remains. Describe the task — the current pipeline, scale, and access constraints. We will come back with an estimate and work plan.