Linux server migration: experience, tools, and zero downtime

Zero-downtime Linux migration: where the move breaks

Migration to another site or a configuration change is usually planned from documentation: services are listed, ports are described, a downtime window is allocated.

In practice, most of the time goes not into data transfer but into discrepancies that surface under production load. The downtime window calculated from the paper plan almost always turns out to be shorter than the real one.

The first failure point during migration is a mismatch in service versions.

The source node could have lived for years on an old branch where a parameter was named differently or had a different default value; on the new node, the same configuration line no longer works or works differently.

A check that "the service is up" does not catch this — it only confirms that the process started.

The second is hidden service dependencies. A reverse proxy calls an internal address that on the new site belongs to another node; an application waits for a mounted directory with shared files; a scheduler job accesses a neighboring host and fails silently.

These connections are not in the service's own configuration, and a package inventory does not show them.

The third is manual configuration edits. A change was made directly on the live server to close an incident and was not recorded in the repository; when deploying from the template, the behavior comes out different, and the cause of the discrepancy is searched for hours.

We start a Linux migration by capturing the actual state — and then we look at how this is closed further.

What changes after migration: timelines, failures, load

The metric by which a migration is considered successful is the unavailability window. In a direct migration, the service is shut down for tens of minutes: take a dump, deploy, switch DNS, and wait for resolver caches to expire.

Managed migration removes this — the database catches up via replication, traffic is moved by switching a virtual address, and the window shrinks to a few seconds.

Switchover time stops depending on data volume. Previously it grew linearly: the larger the database, the longer the downtime. Now data is synchronized in advance, and at the moment of cutover all that remains is to wait for zero replica lag and drain residual writes. The difference is between a scheduled night window and a switchover during the workday.

The share of failed deployments falls not because engineers are more careful, but because environment drift goes away. Configuration is described declaratively and applied identically across all sites, images are built once and promoted across environments without rebuilding.

If before the migration some deployments were rolled back with the phrase "it worked on test," then afterward such rollbacks simply do not reproduce: the environment stops being a source of randomness.

Recovery time becomes predictable. Instead of "bring up the failed node and remember what was changed by hand on it," rollback goes to the previous image or snapshot — minutes instead of hours, with a reproducible state.

What also helps here is that automation roles and playbooks in Ansible live in a repository, not in the heads of on-call engineers.

In terms of load, the migration adds headroom: horizontal scaling no longer hits a single node, and traffic distribution between sites makes it possible to survive a failure without degradation.

The result is not "we migrated," but different baseline operations numbers: shorter windows, fewer failures, predictable recovery.

Migration approaches: full copy, phased migration, hybrid

The migration approach is chosen not by convenience but by two questions: how long the service can be unavailable and how reproducible its state is from configuration.

If the environment is described as code and data is in external storage, the server is rebuilt — the most predictable path, but it requires a full inventory of dependencies.

When state has accumulated manually — local directories, in-place configuration edits — it is faster to take a full copy of the server and then clean up the excess.

Between these extremes, phased migration with intermediate synchronization and a hybrid mode work, where some services already live at the new site and some still at the old one.

In practice, the approach is often combined: what is described as code is rebuilt, what is not described is copied, and synchronization bridges the gap. We fix the choice before work begins — the cutover plan and rollback point depend on it.

Work sequence: from inventory to handover of support

A Linux fleet migration rarely breaks during the transfer itself — failure comes from a connection that was not accounted for in the plan: a forgotten timer in a schedule, the old node's address written in a config, a certificate tied to the previous host.

That is why we build the work sequence so that each next action relies on a verifiable map, not on an engineer's memory.

1. Server inventory. We collect roles, distributions and their branches, kernels, occupied ports, mount points, task schedules, and owners of each node. The output is a registry that shows what is migrated as is and what requires replacement.
2. Dependency map. We record who talks to whom: load balancers, queues, databases, external services via allowlists, shared storages. We separately mark implicit connections — those not described in documentation but surfacing in logs.
3. Target architecture design. We distribute roles across nodes, close unnecessary network access, transfer secrets and certificates, and prepare configuration for automated installation. The architecture is agreed before work begins, not along the way.
4. Environment preparation and testing on a copy of traffic. Target nodes are brought up in advance, and a copy of the real stream with anonymized data is run through them. This catches mismatches in library versions and access rights while the production environment is untouched.
5. Cutover plan and rollback point. We lay out the sequence step by step with windows, owners, and readiness criteria. Each step has a defined rollback point: a data snapshot, a saved config, a reverse load balancer switch.
6. Cutover. Traffic is moved in increments, with pauses to check latency, error rate, and queue metrics. A discrepancy beyond the norm means rollback according to a predefined procedure, without improvisation.
7. Post-migration verification. We compare logs, schedules, rights, responses of adjacent systems, and data integrity. Only after that do we remove the old nodes from the scheme, leaving them available for the rollback period.
8. Support handover. We hand over procedures, the network diagram, the dependency map, instructions for typical failures, and access credentials. The customer's team is on call according to a document, not the memory of an engineer who left.

What you keep: configurations, diagrams, procedures

Documentation is assembled during the work, not reconstructed at the end. Therefore, at handover you receive not a presentation but a working set that lets the infrastructure be maintained without us — by your team or another contractor.

• Configuration descriptions — settings files for web servers, databases, queues, and schedulers with an explanation of why a parameter is set exactly that way and what will break if it is changed.
• Infrastructure diagram — nodes and their roles, network segment boundaries, balancing points, external dependencies, and backups.
• Service interaction diagrams — which service talks to which, over which ports, and what happens when a neighbor is unavailable: where degradation occurs and where failure occurs.
• Access policies and roles — who can read and change what, how keys are issued and revoked, access lifetimes, and the review procedure.
• Maintenance procedures — what to check and how often, which metric values are considered normal, how packages are updated and security fixes are applied.
• Rollback procedure — a step-by-step scenario for returning to the previous state, including data consistency and the point after which rollback is no longer possible.
• Inventory — a list of servers with their purpose and owners, secret storage locations, and rotation rules.

We verify each artifact together with your engineers: we deploy a test environment from it and make sure the descriptions match reality.

Case study: cluster migration without service downtime

The cluster ran on two sites: production and standby, with data replication and a weighted load balancer between them. Switchovers were done manually in a night window: the service was shut down for 4–6 hours, an engineer moved traffic according to a checklist and compared logs.

Over a quarter, two or three unplanned failures accumulated, and incident response time reached 40 minutes — while they found where the data diverged.

What we changed

The second site was kept in constant replication, and configuration was moved into code: Ansible roles describe the state of nodes, the load balancer, and network filters, so the environment is reproduced from scratch rather than adjusted from memory.

The maintenance window was removed — traffic was moved in increments: 1%, 5%, 25%, 50%, 100%, with a pause at each stage while the error rate and latency were normal.

Before and after metrics

Decommissioning the old site came last: the load was removed from it, for three weeks it ran read-only and received backups, then replication was disabled and the machines were removed.

Cluster migration stopped being an event with a night on-call shift — every stage was tested on a portion of traffic.

How Linux migration differs from rebuilding the environment

State migration is when the environment itself moves along with the data: service units, configurations, rights, certificates, scheduled tasks, accounts. Clean deployment is when the infrastructure is brought up anew from a declarative description, and only data is transferred. The difference is not in neatness but in where risks come from and how downtime is allocated.

The main risk of state migration is accumulated drift. A service was edited by hand for years, some parameters are not recorded anywhere, and a particular setting or traffic filtering rule lives only on the old machine's disk.

A tool like Ansible helps describe configuration, but such dependencies are still pulled out from logs and from service behavior when access is disabled.

Clean deployment assembles the same services from a description: roles, configuration templates, schema migrations. Startup takes longer, but afterward maintenance complexity drops — the state is reproducible rather than restored from memory.

The main risk here is service version compatibility: a new set of packages and configuration formats may not accept old data without an intermediate migration step.

In practice, the choice is hybrid. The periphery — load balancer, application runtime, orchestration — is deployed cleanly, while databases, queues, and file storages are migrated with compatibility checks and reconciliation.

We calculate downtime by the longest component, not the average, and keep rollback to the old site available until the change is confirmed under real load.

What is clarified before the start: doubts and answers

By the start of migration, the set of services, mount points, and external dependencies are already described — questions remain whose answers determine exactly how the work will proceed.

Below are the things most often clarified before the start: downtime, the fate of manually assembled environments, and how to make sure that after the migration it is not "roughly the same" but the same thing running.

We separately discuss reversibility: each stage has a rollback, the old environment remains operational until the result is confirmed, and changes are frozen for the duration of the window according to an agreed list.

Some of our services were deployed by hand — will they survive the migration?

Yes, but first they are inventoried: manual edits are extracted from the system and described in configuration.

That way an already deployed environment is reproduced on the new environment one-to-one, not "however it turns out," and stops depending on who fixed something on the server and when.

How do you verify that the result is correct?

By reconciling data, the response of each service, and checking application scenarios against a checklist. Matching checksums and identical responses on the old and new environments are the acceptance criterion, not a general impression that "it seems to work."

Are there backups and reverse migration?

Backups are taken before each stage and tested for restore, not just created. Reverse migration to the old environment remains available throughout acceptance, so an error can be rolled back without rebuilding the system from scratch.

How will this affect developers' work?

The impact is limited to the cutover window: branches, test environments, and access remain, only connection addresses change. The change freeze is agreed in advance so the team plans releases around the window instead of learning about it after the fact.

Let's discuss the task and come back with an estimate and plan

To make the estimate substantive rather than "it depends on complexity," we need a short snapshot of your environment at the input: distribution and kernel branch across the machine fleet, partition and filesystem layout, boot method, and what runs on top — web servers, databases, queues, job schedulers.

Plus constraints: acceptable unavailability window, data placement requirements, hardware or license lock-in, and whether a standby environment exists.

Then we assemble this into a migration plan and come back with specifics:

• scope of work: what we migrate as is, what we repackage into configuration, what we decommission;
• wave-based migration plan — with order, checkpoints, and readiness criteria for each wave;
• timeline estimate in business days: preparation, rehearsals on an environment copy, production windows;
• rollback scheme: which snapshots and copies remain available and how quickly we return to the original state;
• list of risks under your constraints — driver compatibility, repository state, dependencies between services;
• technical discussion with your engineers: review of the plan, edits from your side, agreement on the window;
• handover for support: documentation, accesses, response procedures, if we continue to provide support.

Describe your current environment and constraints in free form — we will come back with migration options, timelines, and a list of work. There will be no imposed stages: you will immediately cut some items, and the rest will form the basis of the work.