Why in-place upgrade Linux fails midway
In-place upgrade Linux rarely fails at the beginning — the machine holds out until the package manager starts resolving dependencies of the new release on top of installed packages.
Repository metadata has already been replaced, some libraries have been updated, but the transaction cannot be closed: a dependency conflict arises, where half the environment pulls the old branch and half pulls the new one. After that, the package manager refuses to work with either set.
The next layer is orphaned packages. The release excludes some components, and they remain without an owner in the new metadata: they are not removed, not updated, hold old shared libraries, and block dependency resolution for their neighbors. They are not in the manually installed list, so during analysis they are not visible.
Outdated configuration files break service startup already. The package manager does not silently overwrite changed settings — a new file is created alongside, and the daemon reads the one that was picked up first.
The parameter format in the new release is different: the service crashes on start or comes up with empty values.
The worst case awaits after reboot: the kernel is new, but the boot image and modules remain from the previous release. The root file system does not mount — this is the unbootable system that is restored manually from a rescue medium, losing hours of service availability.
All these failure points are predictable and are checked before reboot. We'll break down how this is closed — from package inventory to a control run on a copy.
What changes after a controlled system update
The transition between major versions stops being spontaneous: services are restarted in a specified order, and the downtime window stays within the declared interval, rather than stretching into a series of emergency night fixes.
Checks are run on a production copy, so dependencies between services and the restart sequence are worked out in advance.
An in-place Linux upgrade proceeds without reinstallation: packages, configurations, and data remain in the same system, roles are not redeployed. This removes the main risk — the discrepancy between what is configured manually and what is described in the configuration.
A whole class of failures related to outdated libraries goes away: the distribution stops receiving security fixes and stops building against current dependencies.
After moving to a supported Debian branch, builds are reproduced again by standard tools, without patches from third-party repositories.
The rollback plan is fixed before work begins: a state snapshot, a return point, and the criteria under which the transition is stopped. If a service does not pass checks, returning to the previous image takes minutes, not hours of recovery from a backup.
The output is support for current libraries, a repeatable procedure for the next transition, and a predictable recovery time that can be included in a service level agreement.
Methods of transition between major Linux versions
The choice between an in-place upgrade, migration to a clean system, and a parallel staging environment is determined not by engineer convenience, but by how much downtime the service allows and how manageable the configuration remains after the transition.
An in-place upgrade preserves data, local settings, and addressing, but drags along the layers of previous minor versions — from outdated configs to packages that no one dares to remove.
Migration to a clean system gives a predictable state, but requires a full inventory of services and data transfer with integrity verification.
| Option | When it fits | Method limitations |
|---|---|---|
| In-place upgrade | Single node, a downtime window is acceptable, configuration on a managed branch | Residual files and outdated modules are carried over automatically; rollback after the first boot is non-trivial |
| Migration to a clean system | Reproducible builds are needed and manual edits must be eliminated | Downtime during copying; requires a review of all services, access rights, and mount points |
| Parallel staging environment | Load is distributed, there is a load balancer and replicas | Double maintenance during the migration period; discrepancies between environments are harder to debug |
In practice, one starts with an in-place upgrade on a single non-critical node: this is how scenarios, scripts, and service behavior are tested before applying them to production machines.
Data preservation in any option relies on a snapshot before work and a comparison after — otherwise there will be nothing with which to compare “before” and “after”.
How the upgrade proceeds: stages of engineering work
An upgrade between major versions is not a single command in the terminal, but a sequence where each step leaves a path back. Work proceeds one host at a time: state is recorded before and after, so a failure at any stage does not turn into service loss.
The order is the same for distributions in the Debian family and those compatible with it — only the set of package manager commands differs.
- Inventory of services. We collect the host's roles: active units, timers and scheduled tasks, listening ports, connections with neighboring machines, builds bypassing the package manager. Without this list, it cannot be claimed that nothing was lost after the upgrade.
- State snapshot. We record the list of packages with versions, the contents of the settings directory, the mount table, network parameters, installed kernel modules. Plus a full disk image or volume snapshot — it is needed both for comparison and for rollback.
- Rollback plan and maintenance window. We define the point of no return: up to which step rollback takes minutes, after which it takes hours. We agree on the window, access, and the person responsible on the customer side.
- Run on a copy. We deploy a staging environment from the snapshot and go through the entire chain: updating repositories, changing the release, merging configs, rebooting. This reveals in advance packages that have disappeared from the new repositories and changed configuration file formats.
- In-place upgrade. We move services into maintenance, bring packages up to the new release, and resolve configuration conflicts manually rather than by automatic replacement. Reboot goes through the management console so that access does not depend on a working network.
- Post-reboot check. We look at units in a failed state, boot logs, mount points, certificate expiration dates, and application checks: does the service respond, is writing to the database happening, do neighboring nodes see it.
- Acceptance of the result. We hand over a checklist of checks, updated documentation, a list of changes, and an observation window during which we remain in contact. The host is returned to operation based on a confirmed result, not on the fact of reboot.
Upgrade artifacts: configurations and procedures
An upgrade between major versions via in-place upgrade ends not with a reboot, but with the handover of a set of files and documents.
With it, the system can be maintained, reproduced on another staging environment, and rolled back without the participation of the person who upgraded it. Below is the list that we close before handover: if an item is missing, the work is not considered complete.
- Infrastructure diagram — nodes, their roles, network segments, and connections between services, current as of the handover date, not historical.
- Service configurations — settings, systemd units, and kernel parameters that differ from default, with an explanation of why exactly this value was chosen.
- Access policies — group composition, privilege escalation rules, keys and certificates, their validity periods, and the revocation procedure when an employee leaves.
- Maintenance procedure — frequency of updates, maintenance windows, pre-application checks, and the criteria by which a node is recognized as operational.
- Rollback procedure — remaining return points, what exactly is restored from backup, how long it takes, and who makes the decision.
- Change log — chronology: what was changed and why, which checks were passed, what deviations were found, and how they were closed.
The diagram, configurations, and log are stored together, in a version control system, so edits go along with changes, not as a separate document. Each item is confirmed by a file or command — the set is accepted formally, not verbally.
Case: upgrading a server fleet without prolonged downtime
A fleet of 60+ servers was running on a major distribution version for which the vendor had stopped releasing security updates.
The upgrade was postponed year after year: the standard scenario involved reinstalling from scratch, transferring data, and manually configuring each machine — by the team's own estimate, this produced at least two days of downtime per group and dozens of manual operations where it is easy to make a mistake.
How the load was migrated
The fleet was divided into groups of 8–10 machines and migrated sequentially. A node was removed from load balancing, active sessions were allowed to finish, a snapshot was taken, and an in-place upgrade was launched via managed Ansible playbooks.
After reboot, the machine underwent automatic checks: service response, port availability, integrity of data directories.
Only after fully passing did the node return to rotation; otherwise, it was rolled back from the snapshot within minutes, and the remaining machines in the group continued to carry the load.
Metrics before and after
| Metric | Before | After |
|---|---|---|
| Work with one server | 4–6 hours of manual operations | 25–40 minutes, of which 5 minutes are engineer involvement |
| Service downtime | an 8-hour window on weekends | zero, nodes are removed one at a time |
| Share of manual steps | about 70% | less than 10% |
| Rollback on failure | reinstall from scratch | snapshot restore in 3–5 minutes |
| Machines on the current branch | 0% | 100% |
The entire fleet was migrated in a month and a half, without ever stopping the service entirely.
Manual operations were reduced to launching the playbook and checking the report, and an upgrade to the next major version now follows a proven route rather than starting from scratch.
How does an in-place upgrade differ from reinstallation?
An in-place upgrade and reinstallation solve one task — moving the system to a new major branch — but approach it from different sides.
In the first case, the package manager replaces packages on top of the running environment and runs migration scripts: it also compares configuration files, marks manually changed ones, and offers to keep them.
In the second, you get a clean root and transfer data, settings, and services onto it by hand — that is, you reassemble the machine from what you managed to remember and find.
Hence the main practical difference — preservation of settings. An in-place upgrade carries along users and their keys, mount rules, certificates, service descriptions, scheduler jobs, resource limits, and network parameters.
All this remains untouched on disk, and the package manager only asks what to do with files that were changed manually.
Reinstalling the system does not provide this: any setting that was not recorded in the configuration repository or put into a backup disappears along with the old root, and is usually remembered only after launch.
The timeframes differ by an order of magnitude. An in-place upgrade fits into one window: taking a file system snapshot or backup, checking free space, running the upgrade, rebooting, checking services and logs.
Reinstallation is an inventory of what actually runs on the host, image build or partitioning, transfer of data and configurations, access setup, then debugging to a state of “everything as it was,” and this takes days, not hours.
The risk of data loss also differs in nature.
With an in-place upgrade, it is concentrated in one window: an unsuccessful migration, insufficient space on a partition, or an incompatible service configuration — and rollback is done from a snapshot, so a snapshot must exist before the start, and checking on a copy of the staging environment is mandatory.
With reinstallation, data is lost quietly: a database was forgotten, a queue was not exported, a certificate or firewall rule was missed, and this is discovered weeks later.
The practical rule is simple: as long as the system was configured manually and accumulated exceptions, an in-place upgrade is cheaper in time and safer for the state.
Reinstallation is justified when the root is no longer reproducible and it is cleaner to rebuild it from the described configuration. We look at this before starting work and choose the option based on the host inventory, not habit.
Technical doubts before an upgrade
Before any in-place Linux upgrade, the same questions arise: how long downtime will take, whether a system that has not been updated for a long time will cope, and what to do if the machine does not come up. Below are short answers to the ones asked most often.
How long will service downtime last?
The upgrade fits into one maintenance break: package replacement, boot image generation, reboot. Services return to operation after application health checks, not immediately after reboot. In a cluster, nodes are updated one by one, so the external service remains available.
The system has not been updated for several years — will it upgrade directly?
Not always: dependencies, the init system, and configuration locations change between branches. The path is built step by step, through intermediate releases up to the target branch, and before the first step a full disk image is taken — rollback remains possible at any stage.
How is application software compatibility checked?
Before work, the set of services and library versions are checked against the target branch requirements.
What is not supported directly is installed from a separate repository or replaced with a compatible alternative — this goes into the plan in advance, rather than being discovered on the night of the upgrade.
What to do if the system does not boot?
Boot recovery is performed from a separate medium: the kernel or bootloader entry is replaced, changes are rolled back from the taken image. The procedure itself is tested on an infrastructure copy before work, so in production it is performed according to a ready scenario.
Let's discuss your upgrade task
The estimate for an upgrade between major versions rests on environment details: environments that look identical differ in package composition and third-party repositories, and this is exactly what determines the scope of work.
To give timelines and a sequence of actions, rather than a range “from and to,” we need a slice of the environment — all of this can be sent in any convenient form, whether an export from a configuration management system or a text list.
- list of servers and virtual machines with their role in the environment: load balancer, application, database, queue, file storage;
- current distribution versions and the target branch to which you plan to upgrade;
- non-standard package sources, kernel modules, and builds from source — these most often break the transition;
- service configuration files and systemd units that override default behavior;
- dependencies between nodes: shared volumes, replication, external integrations, hard network bindings;
- acceptable unavailability time and the window in which the upgrade can begin;
- what already exists for rollback: snapshots, backups, standby nodes.
Based on the review, we return an environment inventory with marked risk points, an estimate and a stage-by-stage work plan with the order of node upgrades, as well as acceptance criteria — a list of checks that the service must pass before returning to operation.
We fix the handover procedure in advance: what you receive as output, which configurations and rollback procedures remain with your team, and how the system is handed over to operation after completion.
Describe the task — we will come back with an estimate and a plan.