Docker secrets: where keys leak in a container environment
A failure usually starts at the build stage: a password or token is passed into the image through environment variables and build arguments.
The argument settles in the layer metadata, the variable in the final configuration, and anyone with access to the image can look at them: docker history shows the build commands, and the value from the environment is read straight from the running container.
A secret that has landed in a layer stays there even after it has been removed from the instruction — the previous layers do not go anywhere, and you cannot scrub the value out of an already published image, only rebuild it.
The second leak point is the runtime environment. Any process inside the container reads the variables of its own namespace: it is enough to look into the environment file of the first process or to call the process list with variables shown.
The application passes the environment on to child processes, hands it to error reports and telemetry, and if the container engine socket is mounted into the container, the environment becomes visible to all neighbouring services on the node.
The process rights sit right next to it: a container started as a superuser does not separate access, and the secret is read by the same code that accepts external requests.
The third habit is printing the configuration. Build and application runtime logs contain connection strings with the password inside, error output on the first database call, variable dumps at startup.
Then those logs travel into a shared index with a long retention period and broad read rights, and the secret spreads across copies faster than it can be rotated.
Let's break down how this is closed with the built-in docker secrets tools: the value is passed not in the environment, but as a file available only to a specific service.
Isolated access and key rotation: a measurable effect
Secrets in containers live not in the image's environment variables, but in a separate orchestrator store: Docker mounts the value into the container as a file in tmpfs and gives it only to those services that are explicitly allowed it in the configuration. The effect is verifiable by four measurable indicators.
Time to put a new key into operation
Rotation boils down to creating a new version of the secret and restarting only those services that reference it.
Updating one service takes minutes instead of a coordinated downtime window for the whole system; we delete the previous version after the last consumer has re-read the value.
Share of services with segregated access
A secret is bound to a specific service, so the database does not see the payment gateway key, and the web application does not see the queue credentials.
We bring the indicator up to all services: each one has exactly those secrets without which it will not start, and the rest are unavailable to it.
Isolation of environments and a failure class that is gone
Development, test and production take values from different stores — substituting an environment no longer leads to writing into someone else's database.
When one container is compromised, a whole class of failures goes away: leakage through an environment dump, build logs and image layer history becomes impossible.
What stays under control
The lifetime of a key, the list of consumer services and the trace of the previous version — all in the configuration and in the change log. This is how rotation stops being a one-off exercise and becomes a routine operation.
Docker secrets vs environment variables: modes and limitations
Docker secrets hands the value over as a file in tmpfs inside the container: the process reads /run/secrets/<name>, and the environment variable is not stored anywhere.
Environment variables are simpler, but visible in inspect, inherited by child processes and end up in dumps and logs — if the image or a dump leaks, the password goes with it.
| Option | When it fits | Limitations |
|---|---|---|
| Docker secrets | an orchestrator with secrets support, mounting into tmpfs | the application reads a file, not a ready-made variable |
| Environment variables | a local stand, non-critical parameters | visible in inspect and logs, inherited by processes |
| Mounted files | older compose versions, manual management | the secret lies openly on the host, rotation is manual |
| External secret store | multiple environments and clusters, access auditing | an agent and network access at startup are required |
Choosing by environment
On a single node and a stand, a mounted file or environment variables are enough — there is no critical data there. In a cluster, the value is distributed only to the designated services, and a secret update is applied without rebuilding the image.
Choosing by workload type
A long-lived service reads the secret once at startup and keeps it in memory — any mode will do. Short-lived containers and jobs restart often, so rotation matters: with an external store it is automatic, with mounted files it requires a restart.
The limitation of the mode is that the application must accept a file path rather than a ready-made string.
Rollout in stages: from audit to handing over the policies
We start the work by recording the current state: where passwords, access keys and certificates currently lie, who gets them and how.
The stages run in sequence — each relies on the result of the previous one, so at any step it is clear what has already been closed and what has not.
- Secret inventory. We compile a list: what the secret is, which service uses it, where it is stored, how often it changes. The output is a registry with owners and criticality classes.
- Secret flow diagram. We draw the path of the value from the store to the process: mounting into the container, access by service role, rotation. The diagram immediately shows extra copies in environment variables and in images — we remove those first.
- Configuring the store and rights. We create secrets via Docker Secrets, segregate access by services and environments so that the development environment does not see production values. We grant rights on the principle of least privilege.
- Migrating services. We connect them one by one, with a check after each step. This way an error is localized to one service instead of bringing down the whole environment — cost calculated after audit.
- Failure testing. We break things ourselves: revoke access, substitute the value, stop the store — and see how the service survives it.
- Handing over the policies. We hand over the registry, the diagram, the rotation procedure and the on-call instructions.
Testing on a stand and the rollback procedure
We run load and failure scenarios first on a copy of the production environment — with the same roles and rights, but with test values.
Rollback is provided for at every step: until a service has confirmed receipt of the new value, the old one stays valid, so returning is a switch to the previous record, not a restore from a backup.
Configurations, diagrams and policies: what stays with you
After the project is handed over, you are left not with a set of scattered files but with a working secret management system: configurations, a description of the flows and the rules by which the team acts without our involvement. Below is what exactly is recorded and handed over.
- Secret configurations: secret declarations and their binding to services, ways of delivering them into the container, behaviour when a secret is missing — adding a new value becomes an edit in one place rather than a manual operation on the server.
- Flow diagram: where a secret comes from, which environments it passes through, where it ends up during service runtime and where it must never end up.
- Access policies: a "service — secret — access level" matrix, including segregation between environments and roles.
- Rotation policy: frequency, procedure, the person responsible and the point at which the change is recorded.
- Documentation: a description of the architecture, the decisions taken and the deliberate limitations — so that the next team does not reopen them from scratch.
- Instructions for the team: step-by-step scenarios "add a secret", "revoke access", "check the configuration", "roll back to the previous value".
- A list of pre-release checks: what is validated automatically and what stays with review.
Access policies
We describe rights so that a container sees only its own secrets and nothing extra: access is granted to a service, not to a host, and is changed through configuration, without logging into the machine.
Separately, we spell out which people have access to the values themselves — usually a narrow circle, while everyone else works with names and references.
Rotation policy
Rotation is built without downtime: a new value is created next to the old one, the service accepts both, and only after confirmation is the old one revoked.
The policy records the frequency, the person responsible and the procedure for a compromise — when the change must happen immediately and along a different branch of the scenario.
Case: migrating a service's secrets without downtime
A service of three dozen containers lived on environment variables: database credentials, external service tokens and signing keys were distributed in a single file that sat in the repository next to the deployment manifests.
Rotating one value took an evening — editing the file, rebuilding the images, stopping a group of containers and bringing them up again. Over a quarter, this operation was postponed three times: the downtime window did not fit with the product's operation.
Initial state: one value — four copies
The key existed simultaneously in the repository, in the container's environment variables, in diagnostic output and in the application logs on a failed connection.
The value was read by anyone with access to inspecting the container on the host, and rights were revoked by hand: first the service was shut down, then the value was changed, then it was brought up.
The full rotation cycle was four to six hours, a third of the values were reused, and there was not a single automated leak check.
Result: rotation in minutes and testing on the production environment
The values were moved into Docker secrets: the store hands them to the container as a file in a separate memory area accessible only to the process.
Backward compatibility was maintained by double reading — the service first tries the file, then the environment variable; switching is done with a single flag in the configuration, and rollback is done with the same flag back, without rebuilding images.
We tested in steps: a copy of the traffic, then a tenth of the nodes, then all of them.
The connection error rate did not grow, rotation time dropped to minutes, downtime is zero, only one copy of the value remains, and the share of services with an automated leak check reached one hundred percent.
What to do if secrets are already sitting in environment variables
Environment variables are not isolated: the value is seen by any process inside the container, it ends up in diagnostics, in the service description and in the logs at the very first crash.
That is why replacing values is not a single edit but a staged migration, in which the new source and the old one run in parallel for a while.
The move to docker secrets is done via a dual circuit: the code reads the value first from the mounted file, and if it is missing, falls back to the environment variable.
The file lives in a separate memory area of the container and does not settle either in the image or in the run history. The circuit is removed when all instances of the service have completed a full release cycle on the new source.
The order of migration is determined by the impact of an error. Outbound credentials go first — tokens to external services, queue keys: replacing them is reversible and shows up only as integration failures.
Then encryption and signing keys, where consumer synchronisation is required. Last are database credentials, because here an error stops writes.
Leaked values are handled differently from working ones: everything that has been in an environment variable is considered exposed and is not migrated but replaced.
Rotation is done without downtime — first a new value is issued, consumers are switched over, then the old one is revoked, so that rollback remains possible until the end of the migration.
When the last consumer has been migrated, the variable is removed from the service description and a check is put in place that catches its return in diagnostics and logs. After that, the rotation policy works — by schedule and by incident.
Technical doubts: rotation, rollback, environment compatibility
Doubts about secrets are usually the same: will the service have to be stopped for rotation, what to do if the new value turns out to be wrong, and will values surface where they should not be. Below are practical answers to the questions that come up most often.
Can a secret be changed without restarting the container?
Docker Secrets injects the value as a file inside the container, so a running process sees the old version until it restarts. We do rotation in two steps: the application reads the secret on every access, and distribution goes task by task — without service downtime.
How do you roll back a corrupted secret?
We keep the previous value as a separate secret with a version suffix: rollback is changing the reference in the service description and rolling out again. We keep two versions and delete the old value after the new one is confirmed working.
Does this work the same in different environments?
The logic is the same, the source of the value differs: in production it is the orchestrator secret, locally it is a file outside the repository. The application takes the value through a single interface, so only the delivery method changes.
Will secret values end up in the logs?
Only the service account has rights to the file, and diagnostics writes secret names, not contents. A check for value leaks is built into the build, so an accidental log output is caught before release.
Let's discuss your task: a secret circuit for your stack and processes
For the estimate to be substantive rather than "by feel", we need a minimum of inputs about your circuit. Send what you have at hand — we will clarify the rest in conversation and come back with an assessment and a work plan.
- Service diagram: which containers are in the circuit, how they are connected, who reads secrets — the application at startup, the build or scheduled jobs.
- Environments: how many of them (development, test, production) and how they differ in the composition of secrets and access rights.
- Rotation procedure: who initiates it, how often, and what should happen to already running containers after a value changes.
- Current state: where secrets lie now — in files inside the image, in environment variables, in a configuration management system or in an external store.
- Constraints: requirements of the security service and auditors, role-based access segregation, prohibitions on storing in plain text in the repository and logs.
- Rollback procedure: is it acceptable to restart the service when a secret is updated, and within what time this must happen.
- Post-implementation support: whether a rotation policy, team training and on-call support during the first updates are needed.
Based on these inputs, we will come back with a plan: stages, the scope of changes in manifests and rights, the rotation and rollback procedure, and what we check at each step.
After that we stay in touch — helping run the policy and work through incidents until the team starts managing secrets on its own.
Describe your task in an email or a short call — that is enough to begin.