Setting up servers with Puppet: turnkey configuration management

Puppet configuration management: where manual configuration breaks

The Puppet agent receives a catalog from the server at a set interval — a description of what the node should be: packages, file contents, owners and permissions, running services, filtering rules.

Anything not described in the manifests is either invisible to the system or marked as excess and removed. This is where configuration drift begins: a change made by hand on a live server survives only until the next agent run.

An engineer puts out an incident at night: raises the connection limit in the config, adds a task to the schedule, changes the path to the certificate.

The service responds, the task is closed, and nothing made it into the manifests — there was no access to the repository at hand. A few hours later, the agent brings the file back to the reference state and returns the failure, but now only the chat history remembers the cause.

The reverse case is no cheaper: a change on the node does not contradict the catalog, the node quietly falls out of the shared state, and this is discovered at the next update.

Then environments diverge. In the repository, the test and production environments look identical, but in fact they differ by dependency version, the config from a past release, a service that was once disabled manually.

The divergence between environments does not manifest for months — it accumulates and triggers at rollout time, when a setting tested on a stand with a different package set arrives in production. Incidents after an update almost always have this nature, not a code defect.

Below we look at how drift is closed: unified manifests for all environments, monitoring of the actual state of nodes, agent reports on divergences, and a process under which manual changes become the exception rather than the norm.

What a single source of truth provides for a server fleet

Manifests, roles, and profiles describe the state of a node in full: which packages should be installed, which services are running, which files and permissions are on disk.

This is the single source of truth — not a set of commands in someone else's history, but versioned code that passes review before being applied. The actual state of the machine is compared with the description on every run, and any divergence is eliminated automatically.

The practical effect shows at scale. A new node is brought in with the same description as the rest in the group: from preparation to being operational takes minutes, not hours of manual steps and approvals.

The share of manual operations is reduced to one-offs — typical changes, such as accounts, certificates, or service parameters, are distributed with a single apply across the entire fleet.

Release predictability rests on the fact that a change is first tested on a test group and only then goes to production. Rollback is a return to the previous revision of the description, not a restoration from memory.

Fault tolerance grows separately: a node can be recreated from scratch and get exactly the same configuration it had before the failure.

We set up configuration management via Puppet: the state description lives in a version control system, every change goes through approval, and apply reports provide an audit — who, when, and what changed on a specific machine.

The inventory of nodes and their groups remains in the same code, not in a separate table, so the list of machines and their roles does not diverge from reality.

Ways to describe server state: comparison of approaches

Server state can be described in different ways, and the choice of scheme determines how quickly a change from the shared description reaches the fleet and what happens when a node loses connection to the management loop.

Manual configuration holds while there are only a few machines: beyond that, divergences between them accumulate faster than they can be noticed, and recovery after a failure turns into parsing someone else's changes from logs.

Puppet covers all operating modes, so the question is not "what to take in general" but "what to take for a specific fleet." Below are the options that are actually chosen between.

In practice, we rarely deploy just one thing. We set the observation mode without changes as the first step — it shows a divergence report without touching live machines.

Then we enable centralized change application for the main part of the fleet, and leave isolated sites on autonomous nodes, synchronizing them according to a procedure.

Implementation stages: from fleet audit to handover to operations

Implementation begins not with writing manifests, but with inventory: as long as it is unknown what actually runs on the nodes, any role can wipe out a production service. Therefore, the first stages are fact gathering and design, and only then changes on the servers.

1. Fleet audit. We collect the list of nodes, operating systems, package versions, open ports, and actually running services. The output is a map of divergences between what should be and what is.
2. State inventory. We record configuration files, dependencies between services, and "manual" changes that someone made directly. This is the starting point for describing the desired state.
3. Role and module scheme. We break the configuration into roles — load balancer, application, database, cache — and profiles within them. A node gets a role as a whole, modules are reused, duplicates are eliminated.
4. Management loop setup. We deploy the Puppet management server, configure certificate issuance, data encryption, and logging. Every apply is written to a report: you can see who, when, and what changed.
5. Test loop. The stand replicates production in role composition. First a run without applying changes, then with applying — we catch conflicts and verify that a repeated run changes nothing.
6. Pilot on a limited group. We roll out several nodes, measure apply time and service behavior on restart, and practice the rollback scenario to the previous revision.
7. Handover to operations. We expand in waves by group with divergence monitoring, after which we give the team documentation on roles, the procedure for connecting new nodes, access, and the module update order.

Implementation artifacts: modules, roles, procedures, reports

Implementation is complete not when the configuration has been distributed across the servers, but when the customer can change it without us.

Therefore, everything created remains in your repository and under your access — with a description of how to use it and what to do in case of failure.

• Modules and roles — the Puppet module tree is organized by purpose: system-wide settings, web layer, databases, monitoring. Roles assemble ready-made server types from them.
• Manifests and templates — a declarative state description with parameters instead of hard-coded values. A new node is brought up by changing a variable, not by editing code.
• Server fleet documentation — a list of nodes with roles, variables, and dependencies: you can see which change will affect which group of machines.
• Change procedure — how a change enters the main branch, who reviews it, and in what order it goes to the test loop and then to production.
• Rollback procedure — which configuration versions are retained, how to return a group of nodes to the previous state, and what to do if a server has diverged from the description.
• State reports — a regular summary: how many nodes are in the target state, which changes were applied, where divergences and errors are.
• Operations manual — typical operations: add a node, change a variable, remove a server from a role.

Handover includes a walkthrough: we go through each item and show how to make a change and roll it back on your stand. After that, you can maintain the configuration on your own or leave support to us.

Case study: how a class of failures from divergences between environments was eliminated

Starting point

A fleet of several dozen nodes was maintained manually: settings were transferred via checklists, and some changes were made directly in production.

The audit showed divergences between environments in more than thirty positions — package versions, service parameters, directory permissions.

Recovery of a failed node took from three to five hours, and the result each time differed slightly from a neighboring server with the same purpose.

We described the state in Puppet: each node type got a role, a set of modules, and a separate data layer per environment.

Common rules were moved up, while differences between test and production remained only in parameter values — the configuration code became unified for the entire fleet.

What the metrics showed

• Node recovery time — from 3–5 hours to 15–20 minutes: the node is brought up by applying a role, with no manual steps left.
• Share of automated settings — from 55% to 99%; the remainder is licenses and data that we deliberately do not keep in the repository.
• Divergences between test and production — from thirty-plus positions to zero; deviations are now caught by a pre-rollout check, not by the on-call engineer after an incident.

A whole class of failures disappeared, which used to be attributed to "environment quirks": mismatched service parameters, a forgotten package, diverged permissions.

Rollback also became predictable — the configuration is versioned, and returning a node to its previous state takes minutes along with the change history.

How Puppet differs from manual deployment scripts

A manual script answers the question "did the command execute or not," while configuration management answers the question "does the actual state of the server match the description." The difference is visible not on the first run, but on the tenth: the script runs again and either breaks what is already configured or silently skips a step because the file exists.

Puppet first compiles a resource catalog, compares it with what is actually on the machine, and changes only the divergences.

Hence the idempotence of changes: the description can be applied any number of times, and the result is the same.

If the package is already the required version, the service is running, and the config matches the template — no action is performed, and the run completes without changes.

This is what makes it possible to keep dozens of identical servers and not fear a repeated run on a production node.

The order of application is also structured differently. A script executes top to bottom, and all sequencing logic rests with the author: if you did not install a dependency before calling it, you get a failure on a clean machine.

Puppet builds a graph of relationships between resources and determines the order itself, so the configuration is transferred to a new node without rewriting it for that node.

Another scenario where the comparison of approaches is obvious is configuration drift. Someone fixed a parameter by hand while troubleshooting an incident, the script will not know about it and will not correct anything on the next run. A regular run returns the node to the described state and shows exactly what deviated.

When scripts are enough: one to three servers, a one-time setup, no reproducibility requirement, and no second engineer who must get the same result. As soon as there are more nodes and actual state verification is needed, manual scripts turn into a source of divergences.

Common technical doubts before starting work

Before starting, people more often check not the idea itself, but how it combines with what already lives in the infrastructure. Below are the doubts that come up at the assessment stage and what the scope of work depends on.

What about servers that are configured by hand?

There is no need to rewrite them blindly: the state is captured, described, and migrated gradually. We cover a heterogeneous fleet with separate modules for each OS release, and we take nodes with manual configuration under management last, once the rest is already stable.

Are there limitations to the approach?

Yes, and we discuss them before starting: nodes where changes are made only through web interfaces, and deployments with a strict step order, are poorly described declaratively.

We move such places outside the management loop and mark them explicitly — otherwise the configuration will begin to diverge from reality.

What happens after handover?

We hand over the configuration repository, documentation, the change procedure, and training for your team. Support continues for an agreed period: changes, incident analysis, module updates — the transition proceeds without a support gap.

Let's discuss the task: where to start putting things in order

To come back with a timeline estimate and implementation plan, we do not need full rights to the entire fleet, but access to three points: the configuration repository, the machine inventory, and the person who makes infrastructure decisions.

We will gather the rest ourselves — the initial fleet review is precisely about seeing the actual state of the nodes, not how it is described in documentation.

What we need from you to start:

• Inventory: a list of servers, environments, and node groups, including machines brought up manually and not described in code.
• Access to current configurations: manifests, modules, facts. If part of management is already handled through Puppet, we will show what we reuse and what we rewrite.
• Credentials with the minimum necessary rights: read access to the entire fleet and write access to the pilot loop.
• Change window and security rules: what can be touched freely, and what only with separate approval.
• Acceptance criteria: which services and checks we consider the reference, and how we verify that a node has been brought to the target state.

Next comes a pilot loop on several nodes: first a run in check mode without applying changes, comparison of divergences with the current state, then apply and rollback.

Based on the results, we return with a timeline estimate, a wave-based implementation plan — from pilot to the rest of the fleet — and a list of risks.

Describe the task: the fleet scale, what is already automated, and what result is needed — and we will come back with an estimate and a plan.