Building golden virtual machine images with Packer

Creating virtual machine images manually: where it breaks

A typical picture: a server is brought up according to an instruction in the internal knowledge base — disk partitioning, system packages, accounts, access keys, limits, services, certificates.

The instruction is edited by whoever installed it last, so its final form differs from what actually runs in production, and it is no longer possible to reconstruct which steps were performed and which were skipped.

Hence the environment mismatch.

The development stand, test environment, and production were assembled at different times, by different people, and with different assumptions — an error that reproduces consistently in test does not recur in production, and incident analysis begins not with application logs but with figuring out how the environments differ from one another.

Configuration drift accumulates unnoticed: a change is made on a live machine to extinguish an incident faster, it is not entered into the instruction, and after a couple of months the state of the node cannot be described by any single document.

Then the snowball of configurations starts rolling.

Each release and each emergency adds a layer of changes on top of the previous ones; environment reproducibility is lost: a new node cannot be assembled identically, horizontal scaling turns into sequential manual configuration, and rollback becomes a manual operation under load. The cost is hours of downtime and the recurring "it worked in test."

We analyze how this is closed: the image is described declaratively and built automatically, so environments match by composition, not by the engineer's memory.

The effect of a single reference image: release speed and environment failures

Moving to a single reference image changes not individual steps but the order of magnitude.

Releasing a new server stops being a half-day task: the image is built, tested, and stored in a repository, so a node comes up in minutes — and this time hardly depends on engineer workload.

The share of manual operations drops from dozens of steps to configuring what is truly unique for the environment: network addresses, certificates, disk size.

Recovery after a failure stops being a separate scenario.

A node is not restored according to an instruction but recreated from the same reference image — the time to return to service is predictable and does not depend on who is on duty or whether they remember the sequence of actions.

When there is one reference image, the behavior of stands matches, and discrepancies between the test and production environments stop being a source of surprises.

A whole class of failures goes away separately — configuration drift.

Manual edits on a live server, different library versions on neighboring nodes, "temporary" changes that were forgotten, no longer accumulate: any difference from the reference image either goes into the build description or disappears during the next recreation.

Package updates also become manageable — they are tested once and rolled out across the entire fleet.

The reference image is built from a declarative description (Packer and similar), and this description lives in a version control system. A change passes testing in the test environment and only then goes to the fleet.

Hence the measurable result: server release time is measured in minutes, the share of manual operations in units, and recovery after a failure no longer depends on the human factor.

Reference image build options: comparing approaches

The method of building a reference image determines what happens during the next update: creating virtual machine images from a single description yields an identical result, while a manual edit on a live node produces discrepancies. Let us review the options by applicability and limitations.

The fork is simple — where the common ends and the specific begins. The common part (system updates, monitoring agents, access policies) is kept in the base reference image and updated in a single run, while the node role is defined on top of it.

In practice, approaches are combined: the base reference image is updated, roles are built on top of it from the description, and the build itself runs in an isolated environment and does not touch production networks.

How we build the reference image build pipeline

Building reference images is a reproducible process, not manual configuration of one successful machine. We start with inventory and finish with a regulation according to which your team releases new versions without us.

1. Server audit. We reconcile the actual state of machines: packages, services, disk partitioning, network parameters, traces of manual "in-place" edits.

   The output is a map of discrepancies between servers and a list of what should truly be inherited into the reference image and what has long become garbage.

2. Reference image scheme. We fix the base layer: bootloader, network, accounts, monitoring agent, update policy. Role-specific parts — web, database, proxy — are moved into separate layers so as not to multiply monolithic images that later no one dares to update.

3. Reference image build pipeline.

   The reference image description lives as code: a build on declarative templates (Packer) is triggered by a change in the repository, a clean machine is brought up, setup scripts are applied, and the result is published to the image registry with a version and tag.

4. Reproducibility check. Two builds of the same change must match: we compare artifact checksums, clean out floating package versions and calls to external repositories. Until the result is deterministic, the version is not released.

5. Testing. The image is deployed to the test environment, quick role health checks and a rollback check to the previous reference image are performed. If it passes, we mark the version as released.

6. Release regulation. We hand over the document: who initiates the build, how images are versioned, how many recent versions are stored, how to roll back the reference image version with a single operation. After that, your team runs the pipeline.

What we hand over at the end: descriptions, schemes, regulations

Reference images remain not in a presentation but in working form: after handover, the customer's team has everything to release and update images independently, without us.

The kit is designed for an engineer who will open it a month later and repeat the build without correspondence with the author.

• Build descriptions — image templates for each role (web node, application server, database) with the base distribution, package set, setup scripts, and variables for a specific environment. Any edit passes review and goes into the change history.
• Infrastructure schemes — networks, subnets, firewall rules, mount points, and nodes where ready reference images are published. It separately notes how the test environment differs from production.
• Access policies — who can build an image, who has the right to publish it to production, how build keys are issued and rotated.
• Operations instructions — step-by-step procedure: build the reference image, run checks in the test environment, promote to production, and if necessary return to the previous version.
• Update regulations — frequency of rebuilds for security patches, the procedure for testing and decommissioning outdated images.
• Version catalog — which reference image corresponds to which service release, so rollback takes minutes rather than searching for the right file.
• Check matrix — what exactly is tested on the image before publication: boot, network, monitoring agent, absence of extra accounts, and the standard package set.

We hand over the kit in working condition and conduct the first build together with your team — this way documents are confirmed by practice rather than remaining on paper.

Case: fewer environment discrepancies and faster releases

Before reference images, each stand was assembled manually according to an instruction: base packages, kernel parameters, monitoring agents, and accounts were configured in place.

After a few months, the environments diverged so much that the test environment stopped predicting production behavior — some defects surfaced only in production.

Deploying an environment for a task took about a week, and more than half of the time went into approving manual edits and repeated checks.

What was changed

The image description was moved to the repository next to the product code, and we build reference images via HashiCorp Packer.

The image version is pinned in the configuration, a set of checks is run after the build, and any edit to the base configuration goes through review and rebuild — instead of connecting to a live server.

Server recovery was reduced to deploying the latest reference image: parameters are not restored from memory; they are already inside the image.

After several releases, the stands stopped diverging: the same build produces the same result in test and production environments.

Incidents related to configuration discrepancies disappeared from the task flow, and the engineers' freed-up time went into expanding automated checks.

How does a reference image differ from manual server configuration?

A reference image is a machine built from a description: initial setup is performed in advance and fixed in an artifact.

The difference from a manual server is not in neatness but in the fact that the result is reproduced from a single source as many times as needed — the same set of packages, services, accounts, firewall rules, and kernel parameters.

Manual configuration leaves no trace: the configuration lives only on the server itself, changes are made from memory and diverge between nodes with every fix.

Build reproducibility removes this drift — the image is checked by launching a test copy, where checks for service availability, permissions, and the absence of extra open ports are run. An error surfaces before going into operation, not on a production node.

Verifiability changes the cost of maintenance. An edit is made to the image description once, passes checks, and is rolled out to all new machines; in the event of an incident, there is a previous artifact to roll back to in minutes.

A manual fleet does not offer this: two machines can only be compared manually, and the discrepancy is usually found during an outage.

We do not blindly rebuild already running servers.

First we capture the actual state, transfer what is needed into the reference image description, then replace nodes in stages — during planned maintenance or scaling, keeping existing machines in service until the new image is tested.

Common technical questions about building reference images

Before handing over reference image builds, engineering teams ask roughly the same things: how to ensure reproducibility, what will happen during an update, what to do with manually configured servers, and whether they will have to break their already working pipeline. Briefly — on each concern.

How is the reference image updated without downtime?

The new build is prepared in parallel with the running one, and the previous one remains available in the registry for rollback.

Rollout proceeds in groups: first the test environment, then a share of production machines; if metrics deviate, the new instances are removed and the reference image version is returned. The retention rule keeps the latest builds per tag — no need to rebuild for rollback.

Our servers are configured manually — can they be migrated?

We take from the machine a list of packages, services, users, scheduled tasks, and open ports, then transfer this into the image description. Some steps turn out to be garbage from past edits — we do not carry them over.

We bring up the new variant from the reference image alongside, compare behavior with the original server, and only then switch traffic.

We already have our own build pipeline

We describe the image declaratively and embed the verification and publication steps into your accepted build system, connecting existing security checks and a secrets store.

If the pipeline already covers some tasks, we leave it as the core and add what is missing — there is no need to rewrite what works.

Let's discuss the task: what is needed to start work

The first meeting is not for a presentation but to remove uncertainty about the scope of work.

We review which environments are covered by images, whether there is an existing foundation in the form of manual instructions or Packer configurations, and how strict the requirements are for build reproducibility.

Based on the conversation, a framework for the implementation plan emerges: what we start with, what we do in the second iteration, which image checks we build in right away.

Inputs that speed up scope estimation:

• list of target environments — cloud, private infrastructure, local stand — and their share in the total machine fleet;
• today's method of preparing machines: manual installation, scripts, accumulated templates;
• requirements for the base image composition: system environment, packages, monitoring agents, security rules;
• where artifacts are stored and how development and support teams obtain them;
• whether there are releases or migrations with fixed deadlines that need to be met;
• who accepts the result and can allocate time for acceptance and discussion.

We return with a scope estimate, an implementation plan, and the composition of knowledge transfer work: documentation, configuration review, and support for the first independent build by your team. Describe the task — we will return with an estimate and plan.