Developing Kustomize configurations: where manual management breaks down
While there are two environments, manifests are edited by hand: they copy a working set, change the image tag and replica count, and roll it out.
On the third environment, a test stage with a different Ingress and resource limits appears, and it turns out there are no identical files left: discrepancies have accumulated for months, and no one can say which of them are intentional and which are traces of nighttime debugging.
Configuration drift between environments surfaces at the most inconvenient moment. Rolling back to the previous release is not reproducible: edits made directly, bypassing the repository, remain in the cluster.
The history of infrastructure changes lives in chat threads and someone's memory, while Kubernetes declarative configurations exist only on paper — the source of truth is blurred between the repository and the live cluster.
The cost of this model is measured not in minutes per edit but in the class of failures.
An update passed on the test environment but not on production because the sets diverged; a deleted section in the base manifest breaks the neighboring environment that did not override it; a pre-rollout check shows one thing, but the cluster turns out to have another.
Each such situation is investigated manually and ends with yet another manual edit — that is, a new discrepancy.
Developing Kustomize configurations removes the very possibility of such divergence: the base describes what is common to all environments, the overlay only the differences, and manifest assembly is predictable and repeatable. Let's look at how this is addressed in practice.
Operational effect: speed of rollout, rollback, and recovery
Declarative configuration assembly changes how release rollout time is calculated.
An overlay describes only the difference between an environment and the base layer — the developer edits the delta, not a copy of the entire manifest, so the path of a change from branch to running pod becomes short and the same for all environments.
When assembling via Kustomize and storing environments in Git, rollout comes down to applying an already assembled state: there is no step of “manually fixing in the cluster what was forgotten in the repository.”
The share of manual operations in a release falls along with this. Changes to access, limits, replica count, and external service addresses live in separate layers and are inherited predictably, so a typical release proceeds without an engineer accessing the cluster.
What used to rely on the on-call engineer's memory is now recorded in the repository and validated before application.
Recovery time after a failure is determined by whether the system has a known previous state. It does: returning is a commit rollback and re-reconciliation, not emergency editing of objects under load.
Reconciliation returns the cluster to the described state as a whole, including objects that someone changed manually between releases.
The cost of service downtime here is measured not in hours but in fractions of an hour: an incident is closed by switching to a verified revision, not by searching for the cause in the live environment.
A whole class of failures disappears — configuration divergence between stages and prod, which makes the same code behave differently.
Configuration workflows: comparison of approaches
The differences between environments usually fit into a dozen values: replica count, limits, external service addresses, logging modes.
How these differences are described determines the cost of adding a new environment and the predictability of rollout — with one approach, a new overlay is enough; with another, you have to edit all files at once.
Comparison of configuration approaches by three criteria — where differences are stored, how much manual work a new environment requires, and what happens to reproducibility:
| Option | When it fits | Limitations |
|---|---|---|
| Shared configuration base + overlays for environments | Three or more environments, differences are a handful of parameters | Requires discipline: edit either in the base or in the overlay, otherwise environments diverge |
| Independent sets for each environment | Environments have diverged greatly: different components and owners | Every change is replicated manually, drift grows with each new environment |
| Manual edits after generation | A one-off stage for a day or two | The edit is lost on the next generation, reproducibility is zero |
| Templates with variable substitution | Many similar services, values come from outside | Conditions and branches become more complex, validation of correctness is lost |
In practice, the combination “shared configuration base plus overlays for environments” works reliably: the base is validated once, only differences live in the overlay, and review takes minutes.
We leave templates with variable substitution for values from an external source — addresses and credentials; using them to describe the entire configuration means losing validation of correctness and having to parse the generation result manually.
If a project still edits files after generation, that is the first sign that environment description needs to be handled systematically.
Implementation stages: from audit to handover of policies
The work proceeds in waves, and each ends with a verifiable artifact: first we record the actual state, then design the structure, and only then enable automatic synchronization.
This order eliminates a classic mistake — GitOps is connected before configurations are brought to a unified form, and the operator starts deleting what was edited by hand for years.
- Audit of manifests and cluster. We compare the repository with the actual state, find manual edits, duplicates, and discrepancies in limits. The output is a map of environments and a list of resources that must not be given to automation.
- Structure design. We agree on the base layer and per-environment overlays, on naming, on what goes into shared patches and what goes into config and secret generators.
- Setting up the configuration repository. We lay out directories, integrate configuration assembly into change validation, enable schema validation and mandatory diff review before application.
- Assembling environments. Test, pre-production, and production overlays; we make sure differences are described by patches, not file copies.
- Validation and acceptance on the test environment. We run the full synchronization cycle with automatic drift correction, observe behavior during rollback and when a resource is deleted from the repository.
- Production migration. We move the environment step by step, leaving rollback as commit reversion, and close manual access only after stable operation.
- Handover of policies. We provide instructions for adding an environment and a service, change review rules, rollback procedure, and train the on-call shift.
Deliverables: repository, environment schemas, policies, and access
The tangible result is not a scattering of files but a working environment: a repository with Kustomize configurations that deploys from scratch on a clean machine, and the policies according to which your team then changes environments itself. Below is what exactly is handed over to you under the acceptance act.
- Repository structure: base layers, overlays for each environment — development, stage, production, shared modules, and extension points. The order in which layers are applied is fixed, so assembly produces the same result for any engineer.
- Environment schema: a table of differences — what is moved into overlays, what remains shared, what settings are added when a new stage appears.
- Access policies and roles: who edits base layers, who edits only stage overlays, who approves rollout to production. Rights are granted by roles; there are no per-person exceptions.
- Working with secrets: what is stored in a protected vault, how values are substituted into configurations, how rotation works, and why keys do not end up in the repository.
- Change rollout policy: configuration validation before merge, mandatory review by a second engineer, approval procedure for prod changes.
- Rollback: version labels, change history, and a step-by-step procedure for returning to the previous state — including the case when an edit has already affected data.
- Infrastructure documentation: environment map, dependency diagram, from-scratch deployment instructions, and typical operations. A new engineer is onboarded into the environment using it, without verbal explanations.
Case study: how rollout time was reduced and manual edits were eliminated
The project is a platform of 60+ services across four environments: development, test, pre-production, and production.
Configurations were in three places at once: templates with overrides, scripts in the build pipeline, and manual edits by the on-call engineer directly in the cluster when limits needed to be increased or an external dependency address changed.
No one knew for certain exactly how production differed from pre-production — except the engineer who had last changed something.
Before
Rollout of one environment took 40–50 minutes, and most of the time went not into assembly but into waiting and manual checks between steps. In each release, engineers made 10–14 manual edits: some through the cluster control panel, some by editing a file on the server.
An error in such an edit was not reproducible on the test environment — it manifested only in production, and troubleshooting took from two hours. Incidents caused by configuration divergence between environments added up to three or four per month.
After
We moved the descriptions into a single declarative form on Kustomize: a shared base layer, per-environment overrides on top of it, and differences moved into separate files rather than hidden in scripts.
Changes are validated and applied automatically from the repository — the cluster state always matches what is in the main branch.
| Metric | Before | After |
|---|---|---|
| Rollout of one environment | 40–50 minutes | 6–8 minutes |
| Manual edits per release | 10–14 | 0 |
| Rollback to previous state | ~30 minutes | up to 5 minutes |
| Incidents due to configuration divergence | 3–4 per month | 1–2 per quarter |
Worth mentioning separately are long troubleshooting sessions: now divergence is visible at the change validation stage, before the production environment, so the class of failures “it worked on test but not in production” has been eliminated from the process.
How does declarative assembly differ from templating?
Templating substitutes values into the manifest text: the engine renders a single source with variables and conditions, and a valid document is obtained only after rendering.
Overlays — the Kustomize approach — work differently: a patch is applied on top of the shared base, changing individual fields — replica count, resources, external service address, label set. The base remains a standalone working manifest that can be applied as is.
Overlays win where environments differ by a few fields. A strategic merge patch does not rewrite the structure but supplements it, so the shared framework lives in one place, and the stage differences can be read in a minute.
Add a service — it appears in all environments at once, and it can be disabled on a specific stage with a separate exception.
Templates are needed when the structure really branches: different service sets, computed values, generation from an external registry. The price is loss of verifiability before rendering.
A template cannot be applied directly, and an error in a condition surfaces already in the cluster, not on the engineer's machine.
Over the long run, environment maintenance depends on whether the final state can be reconstructed through the chain “base plus patches.” If answering the question “what will go into the cluster” requires opening five templates and three value sets, maintenance turns into archaeology.
The practical choice is a base with overlays as the foundation, and templates selectively, for places with real logic.
Technical concerns: secrets, environments, migration
Engineers' concerns arise not at the level of the idea but at the level of details: where secrets live, what to do with an environment that has been edited by hand for years, and how to move to declarative management without stopping production. Below are answers to what is most often checked before starting.
How are secrets stored?
Passwords and tokens do not end up in the repository as plain text. An encrypted value is stored in git; decryption happens in the cluster using a key available only to that environment.
Rotation is scheduled: the value in the vault is changed, and the overlay rolls the pods without edits to the application code.
We have everything configured by hand — do we need to rewrite from scratch?
No. First we capture the actual state of the environment into manifests and split it into a shared base and overlay. Then comes reconciliation: the assembled configuration is compared with the current one, and the difference is shown before application.
After switching, the environment is managed only from the repository, and manual edits are rolled back automatically.
Can we migrate without stopping production?
Yes, switching proceeds environment by environment: the new variant is brought up alongside the old one, undergoes checks on live traffic, and only then is traffic switched.
Rollback is switching back to the previous overlay; it takes minutes and does not require rebuilding images.
Why do stage and production diverge?
Because differences are edited manually in two places. We move them explicitly into the overlay: replica count, resource limits, external service addresses, image tags. The shared base is the same, so divergence is visible before application rather than surfacing at release time.
What happens after handover?
We hand over the repository, environment schemas, and instructions for typical operations — release, rollback, adding a new environment. Training is conducted on your environments.
After that — support under contract: updates, incident analysis, and overlay refinement as the number of services grows.
Let's discuss the task and come back with an estimate and plan
To make the estimate concrete rather than “plus or minus an order of magnitude,” it is enough to describe a few things in the request — below is what helps calculate the scope faster.
- Target clusters and environments: how many there are, how dev differs from stage and prod.
- Current state of configurations: a single set of manifests, charts, or already assembled overlays; what is duplicated in them.
- Composition of the base layer: workloads, services, routing, environment parameters, secrets, and how they are substituted.
- Access model: who changes configurations, whether separate roles are needed for teams and change review.
- Requirements for rollout and rollback: how quickly to restore the previous state, whether phased rollouts are needed.
- GitOps environment: which branch is considered the source of truth, whether there are restrictions on external repositories.
- Organizational constraints: closed environment, proprietary image registries, security and audit requirements.
On the call, we clarify the boundaries: which environments go into the first stage, what we reuse from current configurations, where separate layers are needed for a team.
As a result, you receive a scope estimate, an order of work by stages, and a scheme by which turnkey implementation of declarative configurations reaches handover of the infrastructure to the team — with documentation and a walkthrough of decisions.
Describe the task using the points above — we will discuss it concretely and come back with an estimate and work plan.