GCP Compute Engine setup: where production breaks
Setting up virtual machines in GCP Compute Engine breaks not at the start, but at scale. The first machines are built by hand: choose a type, add a disk, write an SSH key into project metadata.
While there are five instances and a single engineer manages them, the scheme holds—but then configuration drift begins.
The main reason is that the machine template lives separately from production. A managed instance group (MIG) stores one sample, but firewall rules, labels, quotas, and additional disks are not included in it.
Autoscaling under load brings up a dozen nodes from the template—and they come up different: without database access, with a disk of a different size, without the labels by which the load balancer finds them.
Manual edits in the console are the second mine. Keys in project metadata survive an employee leaving and handover of tasks: access remains, but who is using it is unclear.
OS Login closes this gap, but only when it is enabled at the organization level and tied to roles; enabling it on an individual project, on top of old keys, is pointless.
Configuration problems in production look the same: a node has been recreated, behaves differently, and there is nothing to reproduce it with. Below we break down how this is closed—with a template as the single source of truth, versioning, and checks during recreation.
The effect of managed configuration: speed and resilience
When a fleet of machines is described by templates and policies rather than assembled by hand according to instructions, the economics of operations change.
In Google Cloud, Compute Engine configuration is stored in an instance template and a managed instance group: a change is made once, and during the next update it propagates to all machines without logging in to each one.
The time to bring a machine into service drops from hours to minutes. Network, disks, labels, access, the monitoring agent, and the initial package set come from the template—a new machine comes up predictably identical.
Manual operations decrease by a factor: an engineer works through tasks instead of repeating the same set of settings on every machine.
The resilience of the machine fleet rests on the group: it checks instance health itself and replaces a failed one with a new one from the same template.
This eliminates an entire class of failures—"the machine is alive but configured differently." Key-based access through a single account directory closes the second class: keys are issued and revoked centrally, rather than remaining alive on machines after an engineer leaves.
In our team's practice, discrepancies between machines before the move to managed configuration were found already at the moment of failure—afterward they surface at the template validation stage.
Fleet updates proceed in waves with a halt on deviation, and returning to the previous state comes down to the previous template version.
Deployment options: standalone machines, groups, templates
Deployment options in Compute Engine from Google Cloud differ not so much by machine type as by who is responsible for node recovery. A standalone virtual machine is convenient as a first step: deploy it, configure it, send traffic to it.
But its state lives only on it—manual edits accumulate, and after a failure the node has to be brought up from memory or from an outdated instruction.
A managed instance group removes this coupling: the reference is described in a machine template, and the group maintains the specified number of nodes and recreates failed ones.
Automatic instance recovery triggers based on a health check, but works only where the application does not store unique data locally.
| Option | When it fits | Limitations |
|---|---|---|
| Standalone virtual machine | test environment, rarely changing service, software tied to a specific node | no automatic recovery, configuration drifts from documentation, downtime during restart |
| Managed instance group, one template | stateless services: web tier, API, queue workers | instances are interchangeable, all local data is lost on recreation |
| Group distributed across zones | production with availability requirements | more nodes, a shared state storage layer is needed |
| Group with autoscaling | variable load | scaling is driven by a metric; the template on a live group is not changed—only a new version and planned replacement |
What this means for selection
The machine template is the unit of versioning for the entire configuration. Updating a group means a new template version and planned node replacement; rolling back means returning to the previous version, which has already been validated.
Therefore, the move from a standalone machine to a group is not a rewrite, but transferring already working settings into a template and connecting checks.
Step-by-step setup: from project audit to handover of runbooks
Compute Engine setup goes from inventory to runbooks: building on top of an unaudited project inherits someone else's permissions, orphaned disks, and a configuration that no one dares touch.
The stages below follow the order in which they are performed in a project—each builds on the result of the previous one, not on guesses.
- Cloud project audit. We inventory instances, disks, snapshots, subnets, access rules, and accounts. We remove unused resources and narrow permissions granted "temporarily" to the roles actually needed.
- Infrastructure design. We agree on the number of zones, the composition of managed instance groups (MIGs), and the boundaries between test and production environments—before the first resource is created. The design is fixed as a document against which reality is later checked.
- Template and group assembly. The instance template becomes a versioned artifact: base image, disks, network labels, permission composition. SSH access is through OS Login with corporate accounts instead of keys scattered across project metadata.
- Fault tolerance testing. We forcibly take down a zone and an individual instance: we record how long it takes the group to restore working nodes and whether traffic returns without manual intervention.
- Update and rollback rehearsal. We run rolling instance replacement with a limit on unavailable instances and verify return to the previous template—rollback should take minutes, not require rebuilding the group.
- Runbook handover. We deliver the design, configurations, access policies, instructions for standard operations, and the procedure for failure response, and walk through them with the customer's team.
What you keep: templates, policies, designs, runbooks
The infrastructure remains with you as code and documents, not as an engineer who remembers where to click.
Everything we configure in Google Cloud Compute Engine—machine templates, managed instance groups, OS Login, and access policies—lives in your repository and is deployed reproducibly.
- Machine templates: image, disk type, metadata, health check, and startup script are described declaratively—a new version goes through review like ordinary code.
- Access policies: IAM roles by service account, sign-in through OS Login instead of static keys, separation of production and staging, and a list of exceptions with justification.
- Infrastructure design: network, subnets, firewall rules, load balancer, zone distribution, and quotas—in one file, readable without us.
- Infrastructure documentation: what lives where, which variables are set at input, how to bring up the environment from scratch, and how to tear it down correctly.
- Operations runbook: the procedure for rolling out a new version, rollback, image updates, and access rotation—with an indication of who does it.
- Alerts and dashboards: instance group metrics, service availability, startup errors—linked to the responsible owner.
- Verification log: which scenarios were run (zone loss, instance degradation, disk overflow) and what was confirmed in the test environment.
Your team performs routine releases, scaling, and rollback according to the runbook, without us. We get involved when the architecture changes or help is needed with incident analysis.
Case study: the machine fleet stopped requiring manual edits
A fleet of sixty machines across three zones was maintained by one team, and every change was made by hand: the image was built on site, access was issued with keys scattered across machines, and updates were rolled out one by one through the console.
Replacing a failed instance took an evening—bring it up from a snapshot, patch the configs, return it to the load balancer, check the application.
The worst part was drift: a month after a release, some machines in the zones already differed from the reference, and no one could say exactly how.
| Metric | Before | After |
|---|---|---|
| Machine recovery time | 3-4 hours of manual assembly | 6-8 minutes from the template |
| Share of manual operations when changing fleet composition | about 80% | less than 5% |
| Rollout of a new image version to the entire fleet | two evenings | 25 minutes without downtime |
| Configuration discrepancies between zones | on 12-15 machines | none, configuration comes from the template |
What changed: the instance template became the single source of truth and is versioned in the repository, and the machines themselves are assembled by a managed instance group with zone distribution.
The rolling update policy takes part of the machines out of service at a time, and health checks remove an unhealthy instance and recreate it instead of manual treatment.
Access was moved to sign-in with corporate accounts and roles—keys are no longer stored on machines, and when an employee leaves, the account is disabled in the directory, and access disappears with it.
Rollback is now also an engineering operation, not a search for "that one" image copy: we return the previous template version and start a group update. These are all standard Google Cloud mechanisms, so no custom add-ons appear that need to be maintained.
Manual operations moved from daily work to one-off tasks—approving a new role or editing a template—and that is exactly what reduced the share of manual labor by an order of magnitude.
How does a managed instance group differ from a standalone machine?
A standalone machine is a single instance with fixed parameters that lives until it stops itself or the hardware fails. While the load is steady, the scheme works: predictable configuration, minimal moving parts, everything visible in one place.
The problem is that any reboot, kernel update, or disk failure turns into a manual operation—bring it up, reinstall, restore settings, bring it back into load balancing.
A managed instance group removes this class of work through an instance template and a declared size.
You describe what the machine should look like, how many are needed, and which rule checks health; then automatic recovery recreates a failed node from the template itself rather than "treating" it in place—configuration does not drift over time.
Fleet scaling works as a continuation of the same logic: as the queue or load grows, nodes are added; as it declines, excess ones are removed.
Fault tolerance is achieved not by the reliability of an individual server, but by the fact that losing a node does not change the group's capacity—traffic is redistributed to the remaining ones while a new one comes up.
The cost of the solution is also clear: you need a template, health checks, access through OS Login, and an update strategy so a run does not wipe out the entire fleet at once.
If the load is constant and downtime is allowed by the runbook, a standalone machine is cheaper to maintain. As soon as the requirement "the service must not go down at peak" appears, the group pays for itself on the very first incident.
Technical concerns before work starts
Engineering concerns before starting are not about trust, but about risk: services are already running and serving users, so a configuration error is visible immediately.
We address three questions most often asked when moving to managed templates in Google Cloud, and how we answer them.
Boundaries should be fixed before work begins: what we consider success, what resource headroom we maintain, and who approves changes in production. This is cheaper than revising agreements when half the nodes have already been replaced.
How do you migrate running services without downtime?
A new template version is rolled out to a managed instance group gradually, with a limit on the number of machines replaced at once. Until a node passes the health check, traffic is not sent to it—the old ones continue to respond. Rollback means returning to the previous template version, not restoring from a backup.
What project access do you need?
Roles are granted at the project level, and sign-in is through OS Login with corporate accounts: we do not store permanent SSH keys, and every action is logged.
If policy requires isolation, we work in a separate project or through a dedicated service account with a minimal set of permissions.
What should be done with the legacy of manual settings?
First, inventory: what is configured by hand and what is already described in code. We record manual edits and move them into the template, and we close off direct changes on running machines—otherwise the next rollout will overwrite them and drift will return.
How do we verify that nothing will break?
First, we bring up a copy of production: on it we run the template, access rules, and health checks, and compare metrics and logs with the current ones. Only after that do we launch the first wave of real nodes without touching the rest.
Let's discuss the task: what is needed for an estimate and plan
The estimate for infrastructure work is calculated by the boundaries of changes, not by the number of resources. To come back with an implementation plan and stage breakdown, the project inputs below are enough—we will clarify the rest in writing or on a short call.
- Project structure and environment separation: which environments exist separately, who owns them, and who grants access.
- Network perimeter: subnet topology, egress, load balancers, and connectivity rules between services.
- Images and instance templates: what they are built with, where they are stored, and who updates the base image and how.
- Compute Engine instance groups: zonal or regional, which metrics drive scaling, how health checks are configured, and the machine replacement order.
- Engineer access: how they reach hosts, where role and key accounting is maintained, and whether an access log is needed for audit.
- Current state: what is already described declaratively, what was created by hand, and whether there is a state file and version control.
- Constraints: the allowed change window and downtime, fault tolerance and backup requirements.
If some points do not yet have answers, that is fine—gaps will also be included in the plan as separate stages. Describe the task in words: what works now and what should be the result.
We will come back with an estimate, an implementation plan, and a list of questions that need to be closed before the start.
The next step is a practical conversation: we will align expectations, fix the scope and rollout order, so that the first change in production is predictable.