Spacelift setup: why manual edits break releases
An infrastructure change almost never breaks at the moment of apply—it breaks earlier, when someone edits the production environment directly: through a cloud console, a manual run, or an engineer's local session. Such an edit does not make it into code, but it lives in reality.
The next scheduled apply of the infrastructure description sees a discrepancy and either proposes reverting everything back or silently overwrites someone else's settings.
Configuration drift accumulates unnoticed. Environments diverge: the test one lags several changes behind, production accumulates exceptions, and reproducibility is lost. The error is found at release time—by an incident, not by a pre-apply check.
A separate pain is rollback without history. If it is unknown which change was applied and in what order, the system cannot be returned to its previous state: relationships between resources have to be untangled manually.
Dependent objects—networks, access rules, storage—are recreated or deleted along with their parent, and the release turns into a chain of emergency edits.
Spacelift closes this with process, not discipline: changes go only through code, state is recorded and locked during apply, and every run remains in history together with the plan and result. Below we break down how such a setup is configured and what it delivers in practice.
What changes when infrastructure is described as code
Infrastructure as code moves changes from manual actions into a description that goes through checks and approval before apply.
Change release time drops from days to hours: the change plan is visible in full rather than assembled step by step in the console, so approval is based on facts, not assumptions.
Environment reproducibility becomes a property of the process. The test environment is spun up from the same description as production, so discrepancies surface at the plan stage, not at release time.
A new environment for testing a hypothesis is deployed in minutes instead of days of manual assembly and configuration comparison.
An entire class of failures goes away. Configuration drift—when edits are made directly, bypassing the description—stops being invisible: the pipeline managed by Spacelift compares actual state with the described state and blocks the discrepancy before apply.
This also covers "it works differently for one engineer" and forgotten dependencies between services—they are caught during checks, not during an incident.
Configuration repeatability rests on access rules, network policies, and environment variables being defined once and distributed identically to all environments.
The history of descriptions equals the history of states: rolling back to a previous version is an operation on the description, not a restoration from memory.
In sum, measurable things change: the share of manual operations in releases, the time from edit to production, and the number of failures related to configuration. Each subsequent deployment predictably repeats the previous one—this is what makes infrastructure manageable.
Launch options: managed service, self-hosted runner, hybrid
The automation operating mode is not chosen by feature set—the features are the same. The fork is elsewhere: where the change plan and apply are executed, where the state file resides, and into which perimeter secrets from clouds and repositories go.
Hence the three practical models: a fully managed service, a self-hosted runner inside your network, and a hybrid where approval happens outside and apply happens inside.
| Option | When it fits | Limitations |
|---|---|---|
| Managed service | Resources in public clouds, standard environments, no own platform for runners | Private networks require network connections; access to secrets and logs is partially on the service side |
| Self-hosted runner | Isolation of environments is required, work inside a private network, your own rules for access to secrets | Updates, queues, and node scaling are on you; runner availability is within your responsibility |
| Hybrid model | Plan and approval in the service, apply inside the perimeter; some environments are closed, some public | Two control points: unified policies and state file synchronization between the models are needed |
What to consider when choosing
Look at two constraints: network reachability of target environments and who is allowed to see secrets. If there are many non-production environments and they differ in permissions, environment isolation is set by runner roles rather than separate models. We'll review your access scheme—and come back with an estimate and migration plan.
Spacelift setup: stages from audit to handover
We approach Spacelift setup so that infrastructure changes become predictable: first we analyze the current state, then assemble checks, and only then let the process modify real resources.
Each stage relies on the result of the previous one, and we do not touch production until the scheme has been proven in non-production environments.
- Repository audit. We analyze the infrastructure code structure and find manual edits and discrepancies between configuration and the actual state of resources. The output is a map of what we will bring under management.
- Environment design. We define environments, the order in which changes are promoted between them, and team permissions. We separately record where a change is applied automatically and where explicit approval is required.
- Pre-apply checks. We assemble the mandatory set: preliminary change plan, validation of relationships and rules, and blocking of dangerous operations such as deleting storage. No change reaches apply without passing the checks.
- Trial apply. We run the process in a non-production environment: creating, modifying, and rolling back resources. The actual runtime and places where manual intervention is still required become visible.
- Controlled migration. We migrate production environments one module at a time, starting with the least risky. Each step has a rollback point and a signal by which we roll back.
- Handover to support. We hand over documentation on environments and rules, access, and procedures for standard operations. The client's team performs a change itself—we stay alongside until the first independent release.
What remains with you: configuration, blueprints, procedures
Spacelift setup ends not with a running build pipeline but with a package your team can use to continue working without us. We hand everything over in your repository, with change history and review—not a set of screenshots and verbal agreements.
- Working infrastructure description — declarative files for all resources, split into modules, with comments: why it was done this way and what will break if a block is removed.
- Environment blueprints — which environments exist, how they differ in limits, data isolation, and service composition, and how one environment can be used to spin up a copy of another.
- Access policies — roles and permission boundaries: who can change production, where review is mandatory, and how temporary access is granted and revoked.
- Rollback procedure — what counts as an emergency change, who decides on rollback, and within what time the state is returned to the previous one.
- On-call instructions — step-by-step scenarios for common failures: what to check first, how to read logs, and when to bring in the second line.
- Dependency map — the order of applying changes, where shared resources live, and which actions block the others.
- Process agreements — how review is conducted, how often modules are updated, and who is responsible for each area.
The documents live next to the code, not in a separate folder that becomes outdated within a month: editing a blueprint and editing the description go as one change.
Our acceptance criterion is that a couple of weeks after launch, your engineers make infrastructure changes themselves, referring to the handed-over procedures.
Case study: releasing infrastructure changes without manual edits
The client was a product team with several environments and a dozen services. Infrastructure was edited by hand: an engineer opened the provider console, added resources, and separately reported in chat that it was done.
The context of changes lived in the heads of two people, and releases depended on who was on call that day.
We described the infrastructure as code and moved releases into Spacelift: changes are created as reviewable descriptions, pass automated plan checks, and apply is triggered by role approval.
All environments received the same release order—the discrepancy between the test environment and production stopped accumulating.
Before and after measurements
| Metric | Before | After |
|---|---|---|
| Change release time | from several hours to two days | 20–40 minutes including checks |
| Share of manual operations | most steps | individual confirmations |
| Release incidents | regularly, several per month | a few per quarter |
| Recovery after rollback | hours of manual reversal via chat | return to previous state, minutes |
The biggest gain was predictability, not the launch itself. An engineer sees the full change plan before apply—errors are caught during checks, not in production. Rollback stopped being an emergency operation: the previous state is known and reproducible.
What to do if the infrastructure is already deployed manually
Infrastructure built by hand does not need to be rewritten from scratch—that is the most expensive and risky path. The first step is the reverse: capture the current state and describe resources as they actually exist.
Import reads the actual cloud configuration and generates a description based on reality—discrepancies immediately become visible between documentation, the team's memory, and what is running in production.
Next we deploy a parallel setup. Spacelift connects to the same account, builds a change plan, and shows what it intends to do, but applies nothing. The goal of this stage is to achieve an empty plan: if the description is correct, the difference from reality is zero. We go through discrepancies one by one and fix the description, not the infrastructure itself.
We move to incremental migration, from least risky to most sensitive: first the network, access rules, queues, and auxiliary services, then storage and databases.
Each step is a separate change with a clear rollback point; manual edits in the console are blocked by policy, otherwise the setup will start diverging from reality again.
No need to stop services: the old way of working and the new setup exist in parallel, traffic and data are not migrated, and only the way changes are made changes.
The migration takes weeks and can be interrupted at any point without consequences—it is enough to revert to the previous state description.
Technical concerns: state, secrets, rollbacks
Before starting, clients ask the same questions: where state lives, who sees secrets, and what happens on a failed apply. Below are answers to the point, without generalities.
Where is the infrastructure state stored?
In a remote state storage with versioning and locking during apply. Spacelift does not take ownership of it: it reads, applies, returns it back, and records the version. Two simultaneous runs will not overwrite each other, and the previous state can be retrieved for comparison.
How are secrets and access handled?
Secrets are not stored in the repository and do not end up in logs. Values are injected at run time from an external vault, and access is granted for a short period for a specific run. The reviewer sees what the plan changes, but not the contents of sensitive variables.
How do you roll back an applied change?
A rollback is applying the previous configuration version through the same path as a forward release: plan, check, apply. We enable automatic rollback where it is safe; for data changes, we leave manual confirmation.
Do multiple teams in one repository interfere with each other?
Configurations are separated by directories and policies: each has its own variables, permissions, and apply queue. Shared modules are versioned, so a module update by one team does not silently break another team's plan—the discrepancy is visible at the check stage.
Let's discuss your task: what to send for an estimate and plan
To come back with a timeline estimate, work plan, and migration plan, we do not need an in-depth audit of the entire infrastructure—basic input data on how change delivery is currently set up is enough. The more specific the input, the fewer assumptions in the estimate and the more accurate the plan.
- A repository with the infrastructure description, or at least one working module: it shows the resource structure, naming conventions, and level of reuse.
- A list of environments and the differences between them—what is configured differently in the test environment than in production.
- How changes reach the environment today: manually, via local scripts, or through an existing delivery pipeline.
- A list of external systems the infrastructure works with: cloud, databases, image registry, DNS, secret storage.
- Constraints: maintenance windows, approval requirements, and the division of permissions between teams.
- Known weak points—which operations break most often and how long rollback takes.
From this set, we assemble a work plan: the order of transferring state management, connecting Spacelift to repositories, automated change check points, approval policies, and a migration plan without downtime for production environments.
The timeline estimate comes broken down by stages—so you can start with one environment and expand as your team is ready.
Describe your task—we will come back with an estimate and plan.