Oracle Linux Server installation and configuration: UEK, security, production

Why production on Oracle Linux Server breaks after updates

A scheduled update is the most common cause of failure on a production server, and the culprit is not the patch itself but what it pulls along with it.

The kernel moves to a newer branch, while modules built outside the tree — disk array, encryption, file system, and security drivers — remain for the previous version.

The package manager does not see this: dependencies are satisfied, and the update proceeds without warnings.

It all manifests on reboot — the bootloader selects the new kernel, the set of drivers in the initramfs is incomplete, and mounting a volume or starting a service runs into a missing module. If the module is critical for the root filesystem, the system boots into emergency mode.

The most expensive part is the search. While the on-call engineer looks at application logs and restarts the service, the cause lies in the kernel output: messages about an unsigned or incompatible module, a kernel taint flag, and a SELinux denial.

Every such reboot reduces rollback options — by default, old kernels are purged, and there is nothing left to return to the working branch.

Downtime is counted from the moment of the alert, although in fact it lasts from the night update, and recovery runs into rebuilding modules and manually editing bootloader parameters.

We explain how Oracle Linux Server configuration closes this class of failures: update policy, module signing, the set of retained kernels, and pre-reboot checks.

What leaves production: failure classes and work time

A production failure almost never looks like “the server broke.” It is an incompatible module after a kernel update, lost access after a security policy change, a full log partition, or configuration drift between nodes. Each such case costs hours of downtime and night work for the on-call engineer.

Oracle Linux configured according to a checklist removes not individual symptoms but entire classes of failures:

• kernel updates — the new kernel is kept alongside, and switching and rollback are performed by rebooting into a known working state;
• kernel panic — dump collection is configured and verified in advance, and analysis proceeds from a ready trace rather than guessing at the cause;
• loss of access — mandatory access control and filtering rules are described in configuration, not set manually on a live server;
• partition overflow — log rotation, limits, and alerting trigger before the service goes down;
• node drift — installation is idempotent and repeatable, and drift between servers disappears;
• services not started after reboot — target states are verified rather than “we’ll enable it later.”

The practical effect is measurable: service recovery time drops from hours of manual analysis to minutes — kernel rollback, restart, removal of the restriction.

The share of manual operations during deployment falls by about two thirds, because the procedure is described and reproduces identically on any node instead of being assembled from memory.

This is what production stability means: fewer night calls, predictable behavior after updates, and a clear sequence of actions for the on-call engineer.

Deployment options and server kernel operating modes

The choice of deployment scheme determines the cost of each kernel update. On a single server, the kernel and modules are replaced “in place,” but changing the kernel branch requires a reboot — and the service is unavailable the whole time.

A fault-tolerant Linux cluster on two nodes removes this limitation: nodes are taken out of balancing one by one, updated, and returned, and downtime is reduced to the switchover time.

The price is an arbiter for quorum, state synchronization, and a separate check that the required modules come up after the kernel change.

The kernel operating mode is fixed separately: switching between the primary and compatible branches is first run on staging, and only then rolled out to the pair.

On a single node, it is wiser to keep one verified branch and plan windows — otherwise any kernel change turns into unplanned downtime.

Work sequence: from configuration audit to handover to operations

The work sequence is the same for a single server and for a fleet of dozens of machines — only the depth of the audit differs. First, we record the actual state: no one approves a target configuration from memory, and discrepancies are easier to catch before load is transferred.

Oracle Linux configuration on staging is a separate stage so that changes do not arrive in production “on the fly.”

1. Server configuration audit. We capture the current state: partition layout and free space, active set of repositories and modules, SELinux mode, traffic filtering rules, performance profile, network profiles, scheduled tasks, and service state. The result is a list of deviations from the baseline and risks, sorted by criticality.
2. Target design. We agree on the node purpose, disk and mount point layout, set of services, update policy, and access boundaries. The design goes into a document used to deploy staging and that then lives alongside the server.
3. Staging configuration. We reproduce the target state on a test node: performance profile, partitioning, services, SELinux rules in enforcing mode, crash dump collection, and log rotation. All actions are performed through versioned scripts rather than manually — then production is reproduced exactly.
4. Staging testing and failure scenarios. We run normal operations and emergency ones: disk and network interface failure, log partition filling up, kernel update, and reboot under load. We check whether the node recovers on its own and how long that takes.
5. Go-live. We allocate a window, take a rollback point, transfer load, and stay in touch until metrics stabilize. The update policy fixes what can be installed automatically and what only in a window with verification.
6. Handover with training. We provide instructions for routine operations and incident analysis, walk through access and permissions with the team, and show where to look at logs. We leave when the on-call engineer can handle basic tasks without us.

Configurations, designs, and policies that remain with you

We finish the work not with the fact that “the server is running” but with a set of artifacts your team can use to maintain it without us.

We keep configurations in a repository, and designs and policies as text next to them, so that changing the server and changing the document are one change.

• Configuration files — kernel parameters, limits, disk partitioning, service and logging settings with comments: what was changed relative to the base Oracle Linux Server installation and why.
• Access policies — roles, login methods, key issuance and revocation procedure, address restrictions, and what must be recorded in the audit log.
• Infrastructure design — which services run, how they are connected, where the failure points are, how traffic flows, and what depends on external systems.
• Update and rollback policy — maintenance windows, post-update verification procedure, criteria under which we roll back, and how to restore the previous state.
• Instructions for the on-call shift — typical failures, first actions, which logs to check, whom to involve, and when.
• Server documentation — purpose, owner, dependencies, restart procedure, and post-restart checks.
• Acceptance checklist — how to verify in a few minutes that the environment matches the described state.

Case study: production that stopped crashing after kernel updates

The customer runs a round-the-clock service on Oracle Linux Server. Kernel updates followed the policy, but every third one ended in failure: after reboot, some nodes did not bring up network interfaces, and services returned to operation only after manual intervention. The on-call engineer was woken at night more often than once a month.

What it was like before the changes

The inventory showed that production was updated the same way as the test environment — across all nodes at once — and automatic kernel update installation was not disabled.

The previous kernel was removed during cleanup, so rollback meant restoring the node from an image rather than returning to the previous boot entry.

What was changed and what the result was

Updates were moved to waves: first one node, then groups, each with liveness and load checks. The previous kernel is retained in the bootloader, and a node that does not confirm readiness automatically returns to it. Automatic updates in production are disabled — they remain only in the pre-production environment.

Night wake-ups stopped, recovery no longer depends on who is on call, and the maintenance window shrank. Kernel updates stopped being a risky event and became a planned task.

How does UEK differ from the standard Linux kernel?

The difference is not in the package name but in the kernel branch and in which module binary interface the kernel guarantees.

Oracle Linux ships two options: UEK — its own build on a newer mainline branch, and RHCK — a rebuild compatible with Red Hat at the loadable module level.

UEK is ahead in branch version, so it gets drivers for new network and disk controllers earlier, support for newer CPU instruction sets, and improved scheduler and memory subsystem.

On new hardware, this removes the classic pain: you do not have to look for an external module or wait for the vendor to update the set for the previous kernel.

The price is module compatibility. Drivers built for RHCK will not load on UEK without rebuilding, and built-in modules of DBMS and backup systems are often certified specifically for the RHEL-compatible kernel.

If the software vendor requires a RHEL kernel or the fleet has nodes with external modules, it is wiser to keep RHCK and not break support.

Maintenance also differs. UEK allows security patches to be applied to a running kernel without reboot — the downtime window for closing critical vulnerabilities disappears.

Both branches receive fixes in the same cycle, but on RHCK a kernel update almost always means a planned restart.

For production, the choice is simple: UEK by default if there are no software vendor requirements or external modules; RHCK when compatibility matters more than hardware freshness.

This is tested on staging: boot from each kernel, bring up the modules, capture metrics under load, and record what fails and where.

Technical concerns: security, updates, compatibility

Before starting, almost every team asks the same questions: what to do with an already running server, whether updates will break applications, and how access separation is organized. Below are short answers to the point so that the decision is made on facts rather than assumptions.

Will Linux updates break applications?

Updates pass through two environments: first the test one, where your applications are run and application compatibility is checked by dependencies, then production.

We keep the kernel in a supported UEK branch with security fixes applied on the fly — no reboot is needed for them. Rollback is planned in advance: a snapshot before the update and the previous kernel remain available in the bootloader.

How will access separation be organized?

Each service runs under a separate system user without shell login, privilege escalation follows named rules, direct login as root is closed, and authentication is key-only. SELinux remains in enforcing mode rather than being disabled: policies are extended for your applications. Administrative access goes through a separate node, and actions are recorded in the log.

Let’s discuss the task: what to send for an estimate and work plan

So that the estimate for configuration work is based on facts rather than assumptions, we need a minimum set of initial data. There is no need to send exports and access credentials right away — a description is enough.

• inventory: how many servers, their roles, physical or virtual, and where they are located;
• current state: what is already installed, the Oracle Linux Server branch, whether UEK is used, and whether there is a local package mirror;
• security requirements: SELinux mode, partition encryption, integration with the user directory, and audit rules;
• disk and network design: volume layout, file systems and mount points, SSH access, intermediate nodes, and subnet segmentation;
• updates and operations: how maintenance windows are planned, whether scheduled automatic updates are needed, and how rollback is performed;
• related systems: monitoring, log collection, backup, and configuration automation scripts already in use;
• constraints: unacceptable downtime, deadlines, and regulatory requirements for data protection.

With this set, we will return with a work plan — the order of stages, the list of changes at each, and an estimate of timelines. If some data is missing, discussing the task still makes sense: we will close the gaps with clarifying questions.