Setting up secret rotation: automatic password and key rotation

Secret rotation: where static keys break infrastructure

The same database access key lives in a service config, in the environment variables of the build pipeline, in deployment scripts, and in a backup copy of the configuration on a node disk.

Six months later, the same key ends up in correspondence with a contractor, on a test environment spun up from a production copy, and in repository history, from where it cannot be removed by editing a single file.

The key's lifetime is not set — it equals the project lifetime, and no one can say how many copies actually exist.

A credential leak in such a scheme is not a question of "if" but "when." One export of logs with debug output or an archive sent to a shared file resource is enough.

Then the main property of a static key kicks in: it is single, so any leaked instance is equivalent to access to the entire environment — to the production database, the queue, the artifact storage.

It cannot be revoked selectively: the same shared-password access is configured on dozens of nodes and in external integrations. Rotation turns into a manual sweep, a downtime window, and, worse, a forgotten node that keeps working with the old key and remains an open door.

There is no way to verify that the previous secret is no longer used anywhere — there is no inventory of where each key is applied.

We break down how this is addressed: where to store a key so it never ends up in configs at all, how to set its lifetime, and how to revoke access without stopping the service.

What changes after automating key rotation

The leak window shrinks from weeks to minutes

A secret lives exactly as long as specified in the policy, and is reissued on schedule, not "when someone gets around to it." Even if a key leaks into a build log or an external repository, by the time someone tries to use it, it is already invalid.

Revoking a compromised value is one operation, not traversing dozens of nodes with manual config edits.

Short-lived keys instead of long-lived ones

The application itself requests credentials from the secret store and receives them for a limited term, renewing access before the previous one expires. Restarting services when the password changes is not needed — new values are picked up on the fly, old ones are retired.

Permissions are granted by specific role: one service does not get access to other databases even accidentally.

Manual operations leave on-call duty

Password rotation stops being a task that someone remembers and does by calendar. An engineer describes the policy once — lifetime, frequency, scope — and then rotation happens without human involvement and without a downtime window.

The on-call engineer no longer deals with incidents like "the service went down because someone updated the password in one place and forgot another."

The "forgot to rotate" failure class is eliminated

All consumers of the secret receive it from a single source, so desynchronization between environments disappears as a category. Checks in the build pipeline fail if a secret ends up in code or lives longer than the required term.

This turns a leak from "one day we'll find out" into a measurable event with a known response time — down to minutes.

Key rotation modes: comparison of approaches and limitations

The rotation mode is chosen by two criteria: where the secret physically resides and what serves as the update trigger.

In stores such as HashiCorp Vault, the secret does not end up in configuration — an agent on the node requests it when the service starts, so rotation does not require editing files or manually restarting consumers.

The other options are simpler to implement, but each has its own failure point and its own class of delays.

In practice, modes are combined: the store handles issuance, an event handles unscheduled rotation, and the schedule covers forgotten keys. We document this scheme together with the rollback procedure so that updating a secret does not turn into service downtime.

Stages of implementing secret rotation in an existing environment

Secret rotation does not break production at the moment it is enabled, but when it turns out that a key is read by a service no one remembered: a scheduler, an old worker, an on-call engineer's script.

Therefore, work proceeds from inventory to phased enablement, not the other way around — at every step there must be a way to return the environment to its original state.

1. Access inventory. We collect all places where passwords, tokens, and keys live: application configs, environment variables, build pipelines, backups, integrations with external systems. For each secret, we record the owner, purpose, and current lifetime.
2. Service dependency map. We determine who reads each secret and how: at process startup or on the fly, whether there is a cache, how long the connection lives. Without this map, key rotation causes a cascading failure instead of a planned update.
3. Key issuance scheme. We design a single source of secrets, permission separation by services and roles, and environment separation. Clients do not store values themselves but request them at startup and upon expiration.
4. Configuration and verification on a staging environment. The environment replicates production in terms of service composition. We run scenarios with a deliberately short secret lifetime, observe behavior when the source is unavailable and when several services are updated simultaneously.
5. Phased rollout. We start with secrets that have a single consumer and a low cost of error, then expand coverage. Each step is a separate change with observability and the ability to disable rotation for a specific group.
6. Rollback procedure and handover. We describe returning to the previous value, the on-call engineer's actions during mass authentication errors, and the procedure for adding new services. The scheme, dependency map, and procedure remain with the customer's team.

What remains with the team after handover: artifacts and procedures

Handover does not end with granting access to the store: the team receives a set of artifacts in the repository, each of which can be opened, checked, and reproduced in another environment. Below is what remains in hand and why each document is needed.

• Rotation configurations: rotation schedules by secret type — database passwords, external API keys, certificates — with intervals, a compatibility window, and a note of which services pick up the value without a restart.
• Key flow scheme: from issuance in the store to entry into the process — who reads it, through which nodes, where the value must not appear in logs and build variables.
• Access policies: who can see which secret, who is authorized to trigger rotation manually, who only reads, and what happens when an engineer leaves the team.
• On-call shift instructions: scenarios of typical failures — a service holds an old connection, a new secret did not arrive, a certificate is revoked; what to check and in what order to act.
• Infrastructure documentation: a map of services and dependencies, secret storage locations, failure points, and node owners.
• Node decommissioning procedure: the sequence for revoking keys and deleting secrets so that no active access remains after decommissioning.
• Rotation verification checklist: how to make sure rotation actually happened and was not merely noted in a log — a test call, log review, rollback test.

Case study: reduced the leak window to minutes on a distributed environment

Distributed environment: several clusters, separate environments for development and production workloads, managed over 130 nodes. Before starting, we conducted a secret inventory — where they reside, how many copies exist, who has access.

The picture turned out to be typical: passwords to databases and external services were duplicated in environment variables, configuration files, and partly in repositories.

State before

The database key was rotated manually: coordinating a window, editing configuration on all nodes, restarting services, and checking that nothing fell over — 6–8 hours of engineers' work.

Because of that cost, rotation was performed once a year or less often, and when an employee left, access often remained valid. The leak window was measured in weeks, and no one could say exactly how many copies of the key lived outside the store.

What we changed

Secrets moved to a centralized store on HashiCorp Vault with dynamic credentials: the application receives short-lived access at startup and does not store it on disk.

Policies were set by class — database passwords are rotated on schedule, certificates are renewed automatically before expiration, and external integration keys are reissued together with the release.

The resulting state

• The leak window shrank from weeks to minutes: revoking access is one operation instead of traversing a hundred nodes.
• Manual key rotation as a process disappeared: on 130+ nodes, rotation happens without engineer involvement.
• The number of places where the secret lies in plaintext was reduced to zero — only references to the store remain.
• An employee's departure is handled by revoking the account in minutes, without restarting services.

How automatic key rotation differs from on-demand issuance

These are not alternatives: scheduled rotation and on-demand access cover different risk classes, and on live infrastructure both modes almost always operate simultaneously — a secret store such as HashiCorp Vault can do both.

Scheduled rotation mode is a schedule: a database password, a signing key, a service secret are reissued at a set frequency, and applications take the new value from the store without restarting or editing configuration by hand.

The point is that the window in which a leaked secret remains valid is limited: the old key is revoked immediately after all consumers confirm the transition to the new one.

On-demand access solves a different task — temporary access for specific work. An engineer needs access to the production database while troubleshooting an incident; a contractor needs a key to the test environment for a couple of hours.

Here the secret is issued with a limited lifetime and revoked automatically, and renewal requires an explicit action with justification. Permission separation defines who is authorized to request, who approves, and for which systems this is allowed.

Access audit records who was granted access, when, and for how long: this is what is read during incident review, not only during a scheduled audit.

The practical criterion is simple: permanent service secrets live by schedule, human and one-time access — on demand. If everything is reduced to one mode, you get either manual issuance of long-lived keys or scheduled rotation where access is needed for an hour.

Technical concerns: rollbacks, availability, compatibility

At a technical meeting, concerns are voiced before approval, and that is normal. Below are four questions that we address in the project before launching automatic rotation, so that rotation does not become a source of outages.

How does key rotation affect availability

Rotation proceeds through overlapping two keys: the new one is issued and verified, the old one remains valid until every instance confirms operation with the new one. Nodes are traversed one at a time, with a readiness check after each step.

If a node does not confirm rotation, the process stops and traffic is not switched to it — user availability does not degrade.

The new key did not work — can we roll back?

Yes, and rollback is built into the procedure in advance: the old secret is not revoked until all consumers have run on the new one for at least one cycle. Key versions and revocation times are recorded in the log, and rollback is one operation.

The emergency lifetime of the old key is limited, so a temporary concession does not become permanent.

We work with external contractors

A contractor is issued a personal key with a limited term and a narrow scope of permissions, not shared access to the system. Rotation happens without their involvement, and when the work is completed, the key is revoked with a separate operation.

This way, contractor access does not become a place through which it is then difficult to close entry into the infrastructure.

Let's discuss the task: what data is needed for an assessment

Assessment and implementation plan depend on how accurately we understand the current state. Below is a list of data after which we return not with clarifying questions but with work stages, implementation order, and effort estimates.

There is no need to send the values of the keys and passwords themselves — only the structure and scheme.

• List of services and secret consumers: which applications, background jobs, build pipelines, and external integrations read keys — and which types specifically (database passwords, API tokens, certificates, signing keys).
• Environment scheme: how development, test, and production are separated, where data resides, what access branches exist, and how environments are isolated from each other.
• Current storage method: configuration files, environment variables, a secret manager, or custom scripts — a description and an example of the structure are sufficient.
• Key lifetime requirements: for which types which rotation periods are acceptable and whether there are external restrictions on reissue frequency.
• Rotation windows and acceptable degradation: when services can be restarted, how much downtime is tolerable, whether there are stateful components.
• Roles and access: who owns secrets, who reads them, how the access log is maintained, whether issuance to different teams with different permissions is needed.
• Failure handling and rollback: what happens if the new key was not picked up, whether there is a rollback procedure, and who makes the decision.
• Constraints: what cannot be changed, security and timeline requirements, dependencies on external service providers.

Describe the task in this scope — we will return with an assessment, implementation plan, and stage order.