Chef infrastructure: where configuration automation breaks
A server configured by hand survives until the first emergency. Packages were installed from memory, configs were edited directly on the machine, someone added a firewall rule and did not record it.
Six months later, no one can say how the production environment differs from the test one, and configuration discrepancies surface at deployment time — when it is too late to roll back.
Chef infrastructure removes this uncertainty: server configuration is described by code — recipes, roles, and attributes — rather than a sequence of terminal commands.
The description is stored in a version control system, goes through review, and is applied identically to any node. A repeated run does not change the result if the state already matches the description — this is the foundation on which predictability rests.
Usually it is not the tool itself that breaks, but the discipline around it. Manual edits on a server live until the next synchronization: they either disappear or conflict with the description, and then the run fails midway.
A separate headache is environments: attributes and recipe versions diverge, and the same description produces different results in the test and production environments.
Reproducibility of environments rests on three things: the description is the single source of truth, changes come only through it, and node state is regularly compared against the reference. Below we examine how this is covered and what it gives in practice.
What changes after moving to managed configurations
A unified configuration standard is when server state is described declaratively rather than reproduced from memory or an on-call engineer's notes.
Settings are stored in a common repository, a change goes through review, and is applied identically across all environments — the difference between the test environment and production ceases to be a source of failures.
Server preparation time drops from hours to minutes: a node is brought up from the description rather than assembled by hand.
Recovery time after a failure is no longer determined by how quickly the on-call engineer remembers the sequence of actions, but by how long it takes to run the policy on a clean machine.
The share of manual edits falls: accounts, packages, network parameters, file permissions, and log rotation are configured automatically. An engineer intervenes only where a human is genuinely needed, and each such intervention remains in the change history rather than in chat.
A whole class of failures goes away separately — discrepancies between environments.
The test environment has one library version, production has another; a certificate was updated on one node out of five; a traffic filtering rule was added only where someone remembered to add it. Finding such defects is expensive: the code is formally correct, but behavior differs.
Downtime is also reduced due to predictability: a failed node is not treated in place but rebuilt from the description — this is faster and leaves no traces of manual intervention that hinder the next investigation.
The descriptions are applied by Chef Infra, so the standard is the same for all nodes and is checked before rollout.
Operating modes of the managing server: comparison of approaches
The management mode is chosen based on two factors: how often the configuration changes and whether the nodes have a permanent channel to the center.
With a homogeneous fleet and a release cycle with review, a centralized scheme is sufficient — the Chef Infra managing server stores recipes, attributes, and roles, and nodes pull them on a schedule.
Where the network is broken — segments without inbound access, sites with a narrow channel — direct management gives a false picture: the node is listed in the fleet, but its actual state diverges from the described one. Below are three options for organizing management and their mode limitations.
Comparison of management options
| Option | When it fits | Mode limitations |
|---|---|---|
| Centralized managing node | Homogeneous fleet, stable channel, changes go through review | No connection to the center means no application; a standby node and state recovery are needed |
| Standalone configuration application | Isolated segments, sites without inbound access | The set of roles diverges between nodes, compliance report is collected manually |
| Hybrid scheme with periodic synchronization | Large fleet, some nodes outside the trusted network | Divergence between local cache and the center is detected only during reconciliation |
The hybrid option most often becomes the working one: nodes apply a local copy of the configuration autonomously, while periodic synchronization pulls in changes and returns a report on the actual state to the center.
Site autonomy is preserved, and a single point of truth for the fleet remains — provided that the reconciliation procedure is described and verified.
Chef infrastructure: implementation stages from audit to handover
Implementing Chef Infra begins not with recipes but with an inventory of what already runs in the fleet. The order of steps is fundamental: a skipped accounting step leads to the described state diverging from the real one, and the very first run on production nodes exposes this.
- Server inventory. We collect a list of nodes, OS versions, installed packages, services, open ports, and traces of manual edits. The output is a summary showing how many unique configurations will need to be described.
- Management scheme. We choose the topology: a central server with a recipe repository or autonomous runs on each node. Here we also delineate access, environments, and attribute stores.
- Description of recipes and roles. We assemble typical configuration building blocks into modules, describe server groups with roles, and move environment differences into attributes — the same code serves both the test and production environment by individual calculationRun on the test environment. The environment repeats the production topology, so conflicts of package versions, service startup order, and template errors surface before production, not after.
- Idempotency check. A repeated run should change nothing: the report leaves zeros for changed resources. Any discrepancy means that the node state is described incompletely.
- Handover of support. A repository with change history, instructions for typical operations, training for the on-call shift, and a procedure for changes through review.
After that, the configuration changes only through the repository, not by hand on the server — and the fleet stops consisting of unique machines.
What we hand over: roles, procedures, and artifacts
Configuration automation is valuable exactly to the extent that the team can work with it without the implementation authors.
Therefore what is handed over is not a set of "as it turned out" scripts but a working Chef repository with an explanation of each block and a procedure by which it can be changed.
What the handover includes
- Configuration descriptions — repository structure, purpose of each recipe and node profile, application order.
- Roles and attributes — how nodes are distributed by role, what is overridden at the environment level, where the boundary is between base and application configuration.
- Access policies — who runs the job, who changes the code, who sees node state reports; rights are divided between the infrastructure team, development, and the contractor.
- Secret storage — keys and passwords are moved out of the repository into a secure vault, issued by role, rotation is described, and leakage into logs is excluded.
- Environment schemes — node composition at each test environment, set of roles, switching configuration between environments, and where state is stored.
- Change procedure — the path of an edit: branch, peer review, test environment, production rollout, rollback to the previous version in one run.
- Support documentation — how to connect a new node or role, how to investigate a configuration discrepancy, where to look in the logs.
The handover is closed with acceptance: the customer's engineers themselves make a typical change — a new role or policy edit — and take it through the full cycle to production. If this succeeds without us, support has truly been handed over.
Case study: eliminated configuration discrepancies between environments
The customer is a product team with four environments: development, test, pre-production, and production.
Server settings there were edited by hand according to an instruction, and by the end of the quarter the configurations diverged: the test environment had one set of parameters, pre-production had another, and some edits remained only in production.
They investigated after the fact, when the service behavior in production differed from what had been verified in the test environment.
Measurements before automation: about 40 manual operations per week, a third of them repeated after rollbacks and restarts. Preparing the test environment for a new version took one and a half working days and depended on who exactly was doing it.
Due to discrepancies, 6–8 incidents occurred per quarter, and each pulled a diagnostic chain behind it: first they looked for the problem in the code, then found the difference in the settings.
The description of server state was moved into code on Chef Infra: common recipes for all environments, roles for typical nodes, environment differences — as a separate data layer rather than edits on live machines.
Before application, a compliance check runs that shows which parameters will depart from the reference; a test run on a copy of the node catches an error before production.
The result: 3–4 operations per week instead of forty, the test environment comes up in 20–25 minutes with one run and does not depend on the operator. The number of incidents due to discrepancies dropped to one or two per quarter.
Repeatability is confirmed by the report: parameter sets across all four environments match, and discrepancies are visible before rollout, not after. Rollback to the previous state is the same operation, only with the previous data set.
How does configuration management differ from orchestration?
Configuration management is responsible for the state of each individual machine: packages of the required versions, configuration files, accounts, permissions, running services.
We describe the reference state in code, the agent on the node compares the current state with it and applies only discrepancies — a repeated run changes nothing because there is nothing left to change.
Service orchestration works at a higher level — it is about order and connections between components.
Who starts first, when the database is ready to accept connections, in what order to take nodes out of load balancing during an update, how to distribute secrets and connection parameters to neighbors. What matters here is not an individual server but the consistency of the whole group.
The difference shows up in failure. A node that has received the reference configuration after a failure is operational again — the application will come up even without external management.
The reverse is not true: a launch scenario that ran on an unpredictable machine state will produce a different result on each run.
That is why configuration management lives on its own: it brings nodes to the described form and does so without an orchestrator. Tools like Chef cover exactly this part — bringing to the reference and checking compliance.
The areas of responsibility meet at the rollout of a new version: configuration delivers settings and artifacts, and the orchestrator decides at what moment to restart a node and in what order.
If the reference is not maintained, orchestration turns into a set of manual agreements that cannot be verified.
Common technical doubts before starting work
The doubts here are usually engineering ones: will already running servers connect, how to make sure a recipe will not break production, and is it realistic to start with part of the fleet. Below are short answers; we discuss the details on your infrastructure.
Our servers were configured manually — will they have to be redone?
No. First we capture the actual state: packages, services, configs, open ports. We write recipes so that they bring the node to the target state and do not touch what already matches. Settings not described in the recipe remain as they are until you decide to formalize them.
How to verify changes before production?
First a run on a copy of the role in the test environment, then on one pilot node. Checks describe the expected result: the service is running, the port is listening, the config matches the template.
Production receives the revision only after reconciliation; rollback is a return to the previous one.
Can we cover only part of the fleet?
Yes. Nodes are addressed through roles, environments, and a run list, so we start with one group — for example, the web layer.
There is one limitation: the more heterogeneous the fleet is by system versions and roles, the more branches in the cookbooks and the more important the implementation order.
How are access and secrets handled?
The agent runs under a service account with the minimum required rights; node keys are issued and revoked by Chef Infra Server. Secrets are not stored in the repository — we pass them as encrypted attributes or from an external vault. Each run leaves a report: what changed and on which node.
Let's discuss the task and come back with an estimate and plan
The first conversation is about the task and the current state of the infrastructure, not about tools. We examine which nodes are still configured by hand, where configurations diverge between environments, and how compliance with procedures is currently checked.
Based on the discussion, you receive an estimate and a work plan with stages, roles, and control points.
To avoid guessing, a minimum set of data is needed to start:
- list of servers and environments (development, test, production) and what manages them now;
- access to the configuration repository or an export of the current configuration scripts;
- description of roles: which services, dependencies, and packages are installed on each node;
- compliance requirements — internal procedures, industry standards, what the control service checks;
- how the workflow is organized: who approves changes, how verification before production works, how rollback is done;
- contact of the engineer responsible for infrastructure — they will confirm the details.
The estimate and plan come as a document: stages from inventory to moving nodes under management, scope of work, what we do and what remains on your side, risks and rollback points.
A separate block covers what will change in operations: how configuration is verified before application and how change history is stored.
Describe the task in free form — we will discuss it and come back with an estimate and a work plan.