Configuring GRUB: where production server boot breaks
The most common story: a server reboots after a scheduled kernel update—and instead of booting, it drops into an emergency console.
The cause is almost always in one of three places: a menu entry for a removed kernel remains in the bootloader, initramfs was not built for the new root partition module, or the kernel command line contains a parameter that no longer exists.
The bootloader does not report what exactly it does not like—it simply stops, and then the engineer recovers the system manually, from memory, at night.
The second type of failure is not with an update, but with hardware. After a disk move, controller replacement, or partition cloning, the bootloader loses the root partition: the entry points to an old identifier rather than the current one.
The server is physically healthy, the data is in place, but it is unavailable, and it cannot be brought up without manually editing the bootloader configuration.
A similar situation occurs when kernel parameters were configured on a live server many months ago: no one remembers where the values came from, which part of them is needed for the system to operate, and which is left over from an old task.
Restoring such an environment on a new node from memory does not work—only what is recorded can be reproduced.
A separate cost of these failures is time and predictability. While a server is in the emergency console, it is entirely unavailable: the service on it does not respond, and the on-call engineer is looking for an instruction written by someone else.
Each such story ends the same way: manual boot recovery that relies on a specific engineer's experience rather than a verifiable procedure. Below we break down how this is addressed: what exactly breaks, why, and how to make boot and kernel parameters reproducible.
What managed boot and kernel parameters provide
Predictable reboot
A bootloader with a fixed set of kernel parameters starts the same way during both a scheduled update and an emergency reset. The server enters operating mode without manual edits on the console—a predictable reboot becomes a routine operation rather than a risk window.
Recovery time—minutes instead of hours
If a node does not come up after an edit, the on-call engineer boots from the previous profile and restores the service. Recovery time after a failure drops from hours of diagnostics at the console to a few minutes, and service downtime is limited to one reboot cycle.
A class of failures from configuration divergence goes away
File descriptor limits, memory behavior, and network queue operation are described in one template and rolled out to the entire fleet. Mismatched settings between servers stop being a source of “floating” failures that reproduce only on some nodes.
Changes are applied repeatably, rollback—with one command
Edits to GRUB and kernel parameters go through configuration in a version control system, rather than through manual input on a live server. Rolling back changes means returning to the previous profile revision: after a reboot, the node comes up in a known-working state.
Modes of applying kernel parameters: comparison of approaches
Kernel parameters are applied once—at boot time, and an error in the bootloader line costs you the node: it does not come up or loses network, and access remains only through the console.
Therefore, choosing the application mode is primarily choosing the rollback method, and only then a question of convenience.
Below are five approaches that people actually choose between. They are not mutually exclusive: manual editing lives on the test bench, fragments—on persistent settings, templates and profiles—on the fleet.
| Option | When suitable | Limitations |
|---|---|---|
| Editing the main bootloader configuration file | Single server, one-off debugging | Overwritten by the configuration generator, no change history or validation, edits diverge between nodes |
| Separate fragment file with parameters | Persistent parameters for all kernels on the node | Justified on a small number of nodes, otherwise configurations diverge; the order in which fragments are included matters |
| Declarative templates with configuration management roles | Server fleet, reproducibility requirement | Management infrastructure is required; the template is first tested on one node, otherwise the error propagates to all |
| Set of boot profiles | Different modes on the same hardware: production, debug, isolated | Profiles diverge over time; switching requires a reboot and procedures |
| Network boot for a server fleet with a fallback boot entry | Mass deployment and node recovery | Dependency on the network and boot service; images and profiles must be versioned |
How GRUB configuration implementation proceeds: from audit to handover
The bootloader and kernel parameters manifest only after a reboot: an error in the configuration turns a scheduled restart into an unavailable node. Therefore, we go step by step, and each step has a verifiable result and a rollback point.
- Audit of current boot. Inventory of the fleet: kernel and bootloader versions, where settings reside—locally or in the configuration management system, which parameters are manually overridden. Separately flag nodes with encryption and software RAID—their boot order is non-standard.
- Change plan. Record target parameters and the rationale for each: menu timeout, set of kernels for rollback, memory management, I/O scheduler, disabling unnecessary modules.
- Test bench. A copy of the configuration on the same kernel version: apply edits, reboot, verify that services started normally. Here we also practice returning to the previous kernel from the menu.
- Maintenance window. Agree on the window and apply one node at a time: remove from load balancing, edit, reboot, return to service. The next node—only after the previous one steadily serves traffic.
- Verification reboot. Look not at “did it boot,” but at specifics: whether parameters were applied, what state services are in, whether extra modules came up. A discrepancy—stop and investigate.
- Rollback plan. Configuration copies next to the working ones, previous kernel in the menu, access via the management console. Rollback—edit and reboot, without rebuilding the node.
- Acceptance and handover. The configuration goes into the management system so that new nodes receive it automatically; the customer's team receives instructions and a verification checklist.
What remains with the team: configurations, diagrams, and procedures
After the work, the team is left not only with a configured node but also with a complete set of artifacts, by which the next bootloader edit can be made without archaeology.
None of them is “documentation for documentation's sake”: each covers a specific class of failures—an unsuccessful kernel update, an incorrect parameter, lost access during reboot.
- Bootloader and kernel parameter configuration files—line by line with comments on what each parameter sets and what side effects it has on memory, network, and failure behavior.
- Server boot diagram: partition layout, mount points, mount order, and the location of the boot partition—so that editing one line does not send the system into emergency mode.
- Set of boot profiles: standard, diagnostic with a minimal set of subsystems, and safe with changes disabled—three options for when the standard one does not start.
- Configuration management templates: parameters are moved into managed descriptions, edits go through review and are rolled out identically to all nodes rather than manually one by one.
- Change procedure: mandatory testing on a test node, application window, recording of the previous state, and a pre-defined rollback before reboot.
- Recovery documentation: step-by-step scenario for bringing up a node from scratch, analysis of typical boot errors, and the procedure if access is lost.
- Access matrix: who has the right to edit kernel parameters, who only boot profiles, and how direct access to the bootloader on production nodes is restricted.
- Set of post-application checks: boot, correct mounting, actual kernel parameter values, and service availability—a report is recorded for each node.
Case study: recovery after boot failure without data loss
Before: emergency mode after a scheduled kernel update
A fleet of 60+ servers on Supermicro, the kernel was updated quarterly according to a single procedure.
About every fifth machine did not come up after reboot: the bootloader selected the wrong image, and initramfs was not built for the new kernel—the server went into emergency mode without network.
Recovery of one machine took from 40 minutes to two hours: console access, manual kernel selection in the menu, package rollback. Service downtime in this area—up to 1.5 hours, with some failures occurring during night windows when no engineer is nearby.
What was changed
Switched boot to explicit profiles: fixed the default entry number instead of searching for the “first available” image, set a menu wait timeout, and left the previous kernel as a working rollback point.
Aligned initramfs with the actual module set and root partition, added configuration validation before reboot—the error is detected before the window, not during it.
A separate profile was created for machines with encrypted root and software RAID: that is where failures occurred most often.
Result
Before and after metrics: the share of servers that did not come up after a kernel update decreased from 20% to zero over two quarters of observation; downtime per incident—from 40–110 minutes to 3–5 minutes due to returning to the previous kernel directly from the menu. Boot resilience no longer depends on whether an engineer is on site.
In-place editing or declarative configuration management?
In-place editing wins when there is one server and one parameter is needed: open the bootloader config, add a line, rebuild the boot entry, reboot. On one machine, it is minutes of work and no infrastructure around it.
The question is what happens when there are dozens of such machines and more than one edit has been made.
Kernel and bootloader parameters are an area that package updates regularly rewrite.
When a new kernel version is installed, the installer places a fresh config alongside, and manual lines end up in a file the system no longer reads: the server boots, but with different parameters, and this is discovered during the first failure.
Then there is drift between machines, when nodes that look identical behave differently, and a complete lack of change verifiability: history shows that someone logged in, but not what exactly was changed or why.
Declarative management flips the logic: parameters are described by a template in a role, the role is applied to the entire server fleet, and the node state is brought to the described one on every run.
This is where Ansible comes in—one template per host group covers repeatability of settings, a change is validated before rollout, and rollback comes down to rolling back the template and reapplying the role.
The trade-off—you need to describe the roles once and keep the fleet in a consistent state, including reboot order.
A sensible combination looks like this: debugging and selecting parameters are done manually on one server, and as soon as a value proves itself, it is moved into the role template and rolled out to the rest.
Emergency on-site access remains, but with a rule: any manual edit ends with a commit to the template, otherwise drift returns at the next package update.
If there are only a few machines and they are not connected by a common task, in-place editing is cheaper. If it is a fleet where reproducibility, a consistent state, and a clear change history matter—declarative only, with manual editing as a temporary tool.
Technical concerns: updates, resilience, rollback
Edits to the bootloader and kernel parameters differ from ordinary changes in that the cost of an error manifests before system startup—the machine does not come online.
Therefore the GRUB configuration is stored in a version control system, and access rights to it are restricted to the superuser: an application must not be able to rewrite boot parameters.
What happens if the server does not boot after editing kernel parameters?
The previous working set remains in the boot menu, and the configuration is in the change history. Rollback is one reboot and restoring the file from the repository; access to the management console is kept separate from the network.
Is a server reboot required?
The kernel command line and array assembly rules are read only at startup. Such edits proceed in a maintenance window: first one machine, then the rest in groups.
Can settings be applied only to part of the fleet?
Yes: the configuration is stored as a template with variables by role groups, so databases and application nodes receive different parameters. Rollout proceeds in waves with boot metric checks between them.
How will the changes survive a kernel update?
Parameters are set in the bootloader configuration, not in a separate menu entry. When a new kernel is installed, the resulting file is rebuilt, and the new branch inherits the same parameters—we verify this before rolling out packages.
How do the settings work with software RAID and disk encryption?
Order is critical: the array is assembled, the encrypted volume is unlocked, then control is passed to the root system. The parameters for these steps reside in the boot image—after edits we rebuild it and test on the bench.
Let's discuss your task: how the work begins
Tell us what server fleet needs to be moved to managed boot, what kernels and bootloaders are used, and whether there are requirements for encryption or software RAID.
We will review the current configuration, propose a design with verifiable rollback points and the set of artifacts that will remain with your team. The scope and timeline of work are determined after the audit and agreed individually.