SaltStack configuration: where server configuration diverges
While there are a dozen servers, configuration is kept in people's heads: discrepancies are caught by eye during incident analysis.
With a hundred machines this stops working — someone edited limits manually, someone installed a package from another repository, and these changes never made it back to the repository.
Configuration drift rarely looks like a breakdown. First library versions diverge, then certificate paths, then package installation order — and the environment that yesterday was a copy of production stops reproducing behavior.
Diagnostics turns into manually comparing two live machines, because no reference description of settings exists.
A separate story is the divergence of settings between servers in the same role. One holds cache, the second queue: the configs grew from a common template but diverged on details like the number of worker processes and trigger thresholds.
If the first machine fails, the load moves to the second, and it goes down for a different reason — the investigation drags on for hours.
Automation of server configuration removes the very possibility of such an edit: the target state is described declaratively and applied to a group of machines, not to a specific host.
SaltStack configuration is built on states, pillar, and grains — they record what distinguishes one role from another, and applying shows exactly what changed. Let's break down how this is addressed.
A single source of configuration: what changes in operations
States instead of instructions: configuration stops being a set of commands that two engineers remember and becomes a description of the target appearance of a node — packages, files, running services.
State management is idempotent: a repeated run does not break the server but brings it to the described form, so configuration divergence between nodes stops being the norm.
A typical change is editing one state file: verification in a test environment, then rollout to a group of nodes in minutes.
Previously these were sequential SSH logins to each host with the risk of missing a step; now the time of a typical change is measured in tens of minutes, and the edit remains in the repository history together with the author.
Rebuilding a server from scratch comes from the base state and role, not from a checklist. Server recovery time is reduced to tens of minutes and does not depend on which on-call engineer is on shift: with the same code, the result is the same.
The share of manual operations falls, and with it a class of failures disappears: "the setting is present on one server and forgotten on another." Drift is detected by regular checking and brought to the target form instead of living until the next incident.
Operational predictability lies in the fact that the state of environments is described in one place: SaltStack becomes not "yet another technology" but the point from which you can see how environments differ from each other.
Comparing approaches to SaltStack configuration: operating modes
What exactly we compare
The operating mode determines where the single version of states is stored and where the node gets them from. This affects the point of failure, convergence speed, and how easy it is to catch divergence between sites. Below are three schemes found in practice, with the limitations of each.
| Option | When it fits | Limitations of the approach |
|---|---|---|
| Local run without a master server | Single nodes, isolated segments, initial machine setup | The state base diverges across nodes: there is no overall picture, each server is updated separately, and there is nowhere to view a holistic rollback |
| Centralized master node | A fleet in one site, need a single state base and a common log of applied changes | The master server becomes a single point of failure and a bottleneck as the fleet grows: serving states to remote segments is delayed |
| Hybrid scheme with multiple sites | Distributed fleet, requirements for environment isolation | It is harder to keep state versions synchronized: change coordination, drift monitoring, and a policy for who rolls out an edit first are needed |
The hybrid scheme almost always turns out to be a compromise: it removes dependence on a single master node but shifts the load onto discipline in working with states.
We choose the mode based on the cost of configuration divergence — if divergence between sites leads to different service behavior, saving on the master node costs more.
Implementation procedure: from fleet assessment to handover of policies
SaltStack configuration is not installing agents but transferring manual actions into described states that are applied reproducibly.
An error in one common state spreads across the entire fleet at once, so the order of work here matters more than speed: first inventory and role schema, then state description, verification in a test environment, and phased commissioning.
-
Server inventory. We collect the topology: hosts, their roles, operating systems and services, what is already described and what is configured manually. The output is a fleet map and a list of divergences to eliminate.
-
Role and target group schema. We determine by what attributes states are addressed to hosts and how environments are separated — this determines whether one role can be changed without affecting the rest of the fleet.
-
State description. We move the common parts into base states and environment differences into injectable data. We verify each state for reapplication: the second run must not change anything.
-
Test environment. We deploy a copy of the fleet, where states are first run without applying changes, then with application and rollback to the previous description — this is how errors are caught before production.
-
Commissioning. First we observe divergences without changes, then connect a small group of hosts, followed by the entire fleet. Regular scheduled application captures configuration drift.
-
Handover of policies. We provide documentation: how to add a role, where to change environment data, how to read the application report, and how to roll back when divergence occurs.
What we hand over: states, schemas, and access policies
All configuration changes live in the SaltStack repository, not in chat history and not in the on-call engineer's head.
Therefore handover is not a conversation of "explain how everything works here" but a set of artifacts by which a new person on the team can bring up a server from scratch and repeat any configuration without the author.
What remains in your repository:
- State sets — descriptions of server roles: packages, configuration files, services, and application order. A repeated run does not break an already configured node.
- Master node schema — topology of master and subordinate nodes, trust between them, failure points, and switchover procedure when unavailable.
- Environment separation — where drafts live, where production data lives, and how a state travels between them.
- Access policies and operator roles — who reads states, who edits, who applies on production nodes; permissions are granted on a least-privilege basis.
- Change policy — an edit is verified in a test environment, enters the repository through approval, and only then is rolled out.
- Documentation — repository map, description of variables by environment, instructions for typical operations and failure investigation.
- Recovery procedure — how to rebuild the master node from the repository if it is lost.
After handover, engineers change configuration without us: they add a role, roll out an edit, roll back an unsuccessful change using the log. We remain available for complex cases, but there is no longer any dependency on the author of the configuration.
Implementation case study: metrics before and after on a server fleet
The customer's fleet is about two hundred virtual and physical machines that were configured manually and partly with custom scripts.
Divergences accumulated over years: an edit was made on one node and forgotten to be transferred to the others, while the actual state was checked by walking through servers.
Below is what measurements showed before the implementation case and six months after handover to operations.
Metrics before and after
| Metric | Before | After |
|---|---|---|
| Time to bring a server into operation | 3–4 hours of manual operations | 25 minutes, manually — only node registration |
| Resolving configuration divergence | 40–60 minutes per node | 3–5 minutes for the entire group at once |
| Share of nodes with divergence | about one third of the fleet | isolated ones, detected automatically |
| Number of failures due to configuration per quarter | 9–12 incidents | 2, both outside the managed area |
| Manual edits on nodes | constant, untracked | only through state description |
We worked on SaltStack: node state is described declaratively, the master server distributes it to groups by hardware attributes and role, and divergence is visible by comparing the description with reality.
This delivered the main thing — an edit appears in one place and reaches all machines instead of living in someone's memory.
Handover to operations included a change policy, separation of permissions for state descriptions, and a procedure for verification on a test group before rollout to the entire fleet.
The on-call shift resolves a failure using a clear algorithm instead of restoring settings from memory — that is why the number of failures due to configuration remains low.
What to do if some servers are already configured manually
Manual configuration is not rewritten blindly — first it is described. The actual state is captured from nodes and turned into declarations without applying anything yet. While the description only reads the system and shows divergences, running services are not affected.
Describing the current state begins with inventory: installed packages, enabled services, contents of configuration files, permissions and owners, scheduler entries.
This data is recorded as a baseline — after that every change is checked against it, and it is immediately visible what has deviated from the original form.
It is more convenient to run the transition period in observation mode: the description is run without application, nodes report divergences, but their state does not change. This surfaces rules that conflict with live manual configuration — before they break something.
The gradual transition proceeds by groups of similar nodes, not all at once. At each step, backward compatibility is preserved: previous scripts and manual edits continue to work until the new description confirms that the node state is reproduced from scratch on a clean machine.
SaltStack takes a node under management as the last step, when the description has been reconciled with reality and all divergences are explained.
After that, any manual edit either enters the description or is overwritten on the next run — the node stops being unique, and its state can be brought up again.
Common technical doubts before starting work
Doubts before starting are almost always the same, and they are reasonable. Below are short answers to the point: how SaltStack configuration is structured in terms of resilience, access, and handover of maintenance.
How does managed configuration differ from a set of scripts?
A script executes commands and does not know the current state of the node. A declaration describes the target state and brings the node to it on every run: repeated execution is safe, divergences are visible in the report instead of surfacing at the moment of failure.
How is the master node protected?
Managed nodes authenticate by keys, access to the master node is restricted by an address list and a separate account without interactive login. Secrets are stored outside the repository and distributed on a need-to-know basis.
Can configuration be changed without downtime?
States are applied first to a test environment, then to production nodes one by one — the service continues responding to requests. Rollback is reapplying the previous version, not restoring from a backup.
How are access rights separated?
Roles describe who sees and changes only their own nodes: development does not get rights to infrastructure, the on-call engineer does not get rights to secrets. Every state application is logged with author and time.
What is needed to hand over maintenance to your team?
A state repository, description of roles and environment schemas, instructions for typical changes, and a walkthrough with your engineers. Access is transferred to you; support is needed only during the first edits.
Let's discuss your task: where to start assessing the fleet
To make the conversation specific rather than an overview, we only need the current snapshot of the fleet — from it you can already see which roles are needed, where the bottlenecks are, and how many stages it will take to migrate machines to SaltStack.
- Fleet composition: how many servers, which operating systems and their major versions, where they are hosted — own racks, cloud, or mixed.
- Machine roles: web nodes, databases, queues, cache, load balancers, service servers — with approximate counts for each group.
- How configuration is currently applied: manual edits, scripts, previous management systems, how closely the actual state of machines matches the repository.
- Access and restrictions: how permissions are granted, whether there are segments without external access, where secrets and keys are stored.
- Maintenance windows: when service restarts and node reboots are allowed, which periods must not be touched.
- Rollback requirements: whether result verification on a pilot group is needed and how quickly the previous configuration must be restored.
Then we come back with an assessment and plan: breakdown into stages, pilot group, procedure for migrating the remaining machines, and acceptance criteria.
Starting the project with a pilot makes sense — it reveals the real constraints of the fleet before we touch production nodes. Describe the task in any convenient form, even a draft — that is enough to discuss it substantively.