Infrastructure Automation in Hetzner Cloud: Terraform, CI/CD, and GitOps

Setting Up Hetzner Cloud: Where Manual Management Breaks Down

Setting up Hetzner Cloud through the panel and SSH sessions works exactly until the first discrepancy. Last week we raised limits, added a firewall rule, mounted a disk for logs — a couple of lines in chat, nothing in the repository.

A month later, an engineer deploys a copy for a load-testing environment and gets a different system.

Configuration drift is a direct consequence of manual server administration. Production and test environments diverge on a dozen small things: library version, service startup order, a forgotten kernel parameter.

Each one is harmless on its own; together, they are already a different environment, and any diagnostics begins with checking "what we actually have."

The cost manifests at the moment of failure. Restoring the environment turns into archaeology: rebuilding from memory, finding who changed network rules and when, guessing which certificates are tied to the old address.

Downtime is measured in hours, and no one is confident that the new build is identical to the one that died — a reproducible rollback to the previous state simply doesn't exist.

The third failure mode is the quietest: undocumented changes don't surface until an audit or handover of the project to another team. Then it turns out that half the settings live only in one person's head, and everyone who ever connected has access to the infrastructure. Let's break down how this gets closed.

What Declarative Infrastructure Delivers: Measurable Effects

A declarative environment description removes the main source of outages — the discrepancy between documentation and what actually runs on servers.

While infrastructure is edited by hand through the Hetzner Cloud control panel, every change lives in the history of other people's sessions, and a failure has to be investigated from memory.

A single source of truth about the environment changes the picture: the environment state is a file in the repository, and servers are automatically brought to match it.

Recovery Speed After a Failure

An environment is rebuilt from its description rather than assembled step by step from a checklist.

Returning after a node or an entire zone fails takes tens of minutes: the same configuration that ran before the failure is deployed, including network rules, load balancing, and storage attachments.

Rolling back to the previous state means returning to the previous version of the description, not searching for differences in logs. The class of failures like "forgot a parameter during manual editing" disappears along with the manual edits themselves.

Environment Reproducibility

From one description, test, pre-production, and production environments are spun up — with the same network topology and the same access rules.

Discrepancies between environments stop being a mystery: if something works in test but not in prod, the difference is visible in the description instead of taking days to find.

An environment for load testing or for a separate customer is created and deleted as a whole, without forgotten resources that keep running and demanding attention.

According to industry data from DORA, teams with a high level of automation measure incident recovery in tens of minutes, and it is precisely the speed of returning to a working state that reduces downtime the most.

Infrastructure Management Approaches: Comparison of Modes

Management mode determines how much manual work each change costs and how quickly the team recovers after a failure. Below are four options that teams realistically choose between when setting up Hetzner Cloud.

Limitations of Manual Mode

Manual mode relies on the operator's memory: a server with configuration edits cannot be reproduced one-to-one, so divergence between stands accumulates over months and surfaces at the moment of failure. Recovery after an outage takes hours, and the reason for current settings lives in chat.

The Cost of Moving to Declarative Description

You can transition in parts: first describe the network, access rules, and servers, then image builds and delivery.

The main costs are importing existing resources and a shared state store; after that, every change is checked before applying, and the environment is spun up reproducibly and identically.

Implementation Order: Audit, Setup, Validation, Handover

Implementation in Hetzner Cloud proceeds in iterations: first we record the actual state of the environment, then transfer it into a description, and only after that enable automated checks.

At each step, the environment remains operational — rollback is possible until the new setup has confirmed equivalence.

Audit of the Current Environment

1. Inventory: collect servers, networks, volumes, DNS records, certificates, access permissions, and external dependencies; put discrepancies between documentation and reality into a separate list.
2. Configuration audit: check firewall rules, permission separation, key rotation, backup modes; mark manual edits that will not survive resource recreation.
3. Migration plan: break the transfer into groups — what is described as code immediately, what remains manual, in what order environments are switched; for each step, define a rollback point.

Build, Validation, and Handover

• Environment as code: Terraform modules, environment separation, state in a shared store with locks, review of the change plan before applying.
• Pipeline: image build, tests, publication to a registry, delivery via a GitOps agent; secrets come from a vault rather than being stored in the repository.
• Configuration validation: compare the actual state with the described one and run failure scenarios — node unavailability, loss of connection to the state store, an erroneous plan.
• Handover to operations: release and rollback procedures, monitoring and alerts, incident review process, team training, and access handover.

Implementation Artifacts: Configurations, Diagrams, Procedures

At the end, the customer's team has not a set of screenshots from the control panel but a repository from which the infrastructure is redeployed from scratch.

Configurations, diagrams, and procedures sit next to the application code, go through review, and are versioned — any environment change is visible in history.

• Infrastructure description — Terraform configuration: servers, networks, load balancers, storage, and the connections between them.
• Network diagram — separation into private and public segments, subnets, internet egress points, routing rules.
• Access policies — roles and keys, separation of production and staging environments, the procedure for granting and revoking access.
• Recovery procedure — target recovery time and recovery point, procedure for restoring from snapshots, roles during a failure.
• Environment documentation — how prod, stage, and test environments differ, which variables, secrets, and limits are set.
• Deployment instructions — how build and delivery work, where manual approval points are, how to roll back a release.
• On-call cheat sheet — typical incidents and first actions for each.

Documentation and Recovery Procedure

The procedure describes recovery step by step: bringing up the network, deploying services from configuration, post-rollback validation. It is updated with every infrastructure change — otherwise, in a month it diverges from reality. Once a quarter, we run a training recovery on a separate site and record the actual time.

Case Study: Moving a Service to a Declarative Environment Description

A service made up of six components ran on servers assembled manually in the Hetzner Cloud panel: the configuration of each environment existed only in the action history of the engineer who set it up.

Six months later, the test environment stopped matching production — discrepancies surfaced during deployment, and bringing up a new environment took two to three days and depended on one person who remembered the sequence of steps.

Before and After Metrics

The environment became the result of a single description: change a variable — only it changes, not a set of manual edits scattered across three environments.

Re-running on a clean environment produces an identical result, so "it works on my machine" stopped being an argument.

What Environment Separation Delivered

We separated production, pre-production, and test environments into distinct description sets with shared modules. Divergence is now visible at the change validation stage rather than in production: configuration differences are computed automatically and go into review.

Rollback is performed by reapplying the previous description — a release stopped being an irreversible action, and the on-call engineer restores a working version in minutes, without restoring from backup.

Result: environment preparation dropped from days to half an hour, and the share of releases that required manual rework after deployment fell by a factor of three.

What to Do If the Infrastructure Is Already Deployed Manually?

An environment deployed manually does not require stopping the service to come under description management. The first step is resource inventory: collect servers, volumes, private networks, load balancers, DNS records, firewall rules, and what is actually running on each node.

Part of the picture is visible through the provider's API, and part only from the machines themselves: systemd units, cron jobs, containers, mount points, and external dependencies that only the person who set them up remembers.

Next comes a description of the current environment, not the desired one. We record the current state as is, including manual edits and historically complex places, and import existing resources without recreating them.

Importing existing infrastructure reads real identifiers and binds them to the configuration — the service keeps running, and the change plan at this stage remains empty.

An empty plan is the main criterion: it proves that the description matched reality, and only after that can anything be changed.

Phased migration goes from the periphery to the core: first what does not affect traffic — snapshots, networks, access rules, then environments, and only after that nodes under load.

Each stage is closed with a plan check without applying; we resolve discrepancies down to zero, and only then do changes go to prod.

Separately, we decide what remains manual: stateful storage is introduced later and more carefully, because the cost of error there is higher than for a web node.

The safety net at each step is the same — a fresh snapshot and no changes in prod until explicit confirmation. This way, the environment moves under description without downtime, and further edits go through review like ordinary code.

Technical Concerns: Access, Secrets, Change Rollback

Concerns about infrastructure projects are usually the same: where keys are stored, who has access to the production environment, what happens if a change breaks the configuration.

Below are short answers on each point: how this is arranged in our implementations on Hetzner Cloud and why this way rather than another.

How are roles and permissions separated?

Each role gets the minimum necessary set of permissions: an engineer applies the plan in the test environment, and production changes only through the pipeline after approval.

There is no permanent manual access to production: access is granted for a period and recorded in the log.

What happens with an unsuccessful change?

The infrastructure is described as code, so rollback means returning to the previous revision and reapplying. State is versioned and changes are idempotent, so recovery takes minutes instead of hours of manual fixing.

How is action auditing performed?

Every change goes through a merge request with mandatory review, so history shows who edited the configuration, when, and why. The account action log is stored separately from the working environment.

Let's Discuss the Task: We'll Analyze Your Environment and Come Back with a Plan

To come back with an estimate and work plan rather than generalities, we need access to facts about your environment in Hetzner Cloud. Below are the inputs we request at the start; some of them can be gathered directly on a call if there is no documentation.

• Infrastructure diagram: which services run, where they are hosted, what external dependencies exist, and what is critical for availability.
• How changes are released now: where code and configurations live, who and with what command performs a release, how long rollback to the previous version takes.
• Access for analysis: read-only tokens to the cloud, access to the code repository, service health monitoring systems, and logs.
• Constraints: security requirements, industry regulations, windows during which work is prohibited, and services that cannot be stopped.
• Scope expectations: implementation from scratch, expansion of current infrastructure automation, or infrastructure support after launch.
• Failures that have already occurred: what broke, how recovery was done, which manual operations are performed regularly and by whom.

In response, you receive an environment map with bottlenecks, a phased plan, scope of work, and a responsibility boundary: what we cover and what remains on your side. Describe the task — we'll come back with an estimate and a plan.