Updating the Linux kernel remains one of those operations that makes system administrators check the maintenance window twice. Live patching can fix certain vulnerabilities without rebooting, but installing an entirely new kernel still normally means restarting the host, migrating virtual machines, or accepting some downtime. Kexec HandOver (KHO) and Live Update Orchestrator (LUO) aim to change precisely that part of the Linux sysadmin workflow.

KHO and LUO in 30 seconds

KHO, initially merged into Linux 6.16, can preserve selected memory regions across a kernel transition using kexec.
LUO extends this approach by coordinating the preservation and recovery of resources used by applications.
Google develops luo-agent, which includes the luod daemon and the luoctl administration utility.
One of the most interesting targets is virtualization: changing the host kernel without shutting down its virtual machines.
systemd 261 already includes LUO-related support, although the technology remains young and should not be confused with transparent kernel upgrades for every Linux server.

For a system administrator, the idea can be reduced to something fairly simple: the old kernel disappears and a new one starts, but some of the state required by running workloads manages to cross the boundary between them.

This is not exactly a server that never reboots. Nor is it traditional live patching. A kexec transition takes place, so the kernel really is replaced. The important difference is that changing the kernel may eventually no longer require stopping everything running on top of it.

From kexec and KHO to a completely different kernel

Linux has offered several ways to reduce the impact of kernel maintenance for years.

Ksplice, kpatch and other live-patching technologies can modify certain parts of a running kernel. They are particularly useful when a vulnerability needs to be fixed quickly without waiting for the next maintenance window.

The simplified model looks like this:

Current kernel
      |
      +-- apply live patch
      |
      v
Same kernel + fix

KHO and LUO pursue a different model:

Kernel A
   |
   +-- preserve state
   |
   +-- kexec
   |
   v
Kernel B
   |
   +-- recover state
   |
   v
Continue compatible workloadsCode language: PHP (php)

The mechanism required to jump from one kernel to another already existed: kexec.

Instead of completely shutting down the server, going through BIOS or UEFI, executing the bootloader and starting the operating system again from scratch, kexec can load another kernel directly from the currently running kernel.

The problem is state.

Starting another kernel normally means losing the state associated with the previous one.

That is where Kexec HandOver comes in.

KHO was initially incorporated into Linux 6.16 and provides infrastructure for preserving selected memory regions during a kexec transition.

The kernel needs the corresponding support:

CONFIG_KEXEC_HANDOVER=y

KHO can then be enabled through the kernel command line:

kho=on

There is also kho_scratch, which reserves memory for the process. A test configuration could look like this:

kho=on kho_scratch=16M,512M,256M

The Linux kernel documentation explains that KHO uses a Flattened Device Tree (FDT) to describe state that must be handed over to the next kernel. Preserved memory regions remain available during kexec, allowing the new kernel to locate them.

That solves one part of the problem: moving memory across the transition.

But preserving a few megabytes or gigabytes is not enough to keep an application alive. The new kernel must understand what that data represents and which resources need to be reconstructed around it.

That is where LUO enters the picture.

LUO brings live kernel updates into userspace

Live Update Orchestrator provides infrastructure for coordinating resources that need to survive a kernel transition.

The subsystem uses:

/dev/liveupdate

as an interface with userspace.

An orchestrator creates LUO sessions, and participating applications can associate resources with those sessions before the kernel transition takes place.

Conceptually, the process looks like this:

Application
    |
    +-- register resources
    |
    v
LUO session
    |
    +-- preserve
    |
Kernel A
    |
    +-- kexec
    |
Kernel B
    |
    +-- retrieve
    |
    v
Restored application

LUO defines operations related to preserving, freezing, retrieving and finalizing resources.

This becomes particularly interesting with mechanisms such as memfd.

An application may keep a large region of memory associated with a file descriptor. Instead of writing its contents to disk, destroying it and reconstructing everything after boot, LUO can help preserve that memory across the transition.

It is easy to see why this matters to Google and other large infrastructure operators.

The cloud scenario that matters: QEMU/KVM

Consider a Linux virtualization node running dozens of QEMU/KVM virtual machines.

A major kernel update is released.

A conventional maintenance procedure might look something like this:

1. Stop scheduling new workloads on the node
2. Migrate VMs to other hosts
3. Verify that the node is empty
4. Update the kernel
5. Reboot
6. Validate services and networking
7. Return the node to the clusterCode language: PHP (php)

For a small cluster, this is perfectly manageable.

At hyperscale, the situation changes.

Evacuating hosts requires spare compute capacity, enough network bandwidth for migrations, automation and coordination. The more urgent kernel updates an operator needs to deploy, the greater the operational cost.

LUO is intended to enable another model.

QEMU could preserve resources required by a virtual machine, transition to the new kernel and reclaim those resources afterwards.

The target would look approximately like this:

                    LINUX HOST

              +------------------+
              |       QEMU       |
              |                  |
              |   VM1 VM2 VM3    |
              +---------+--------+
                        |
                  preserve state
                        |
                        v
              +------------------+
              |     Kernel A     |
              +------------------+
                        |
                      kexec
                        |
                        v
              +------------------+
              |     Kernel B     |
              +------------------+
                        |
                   recover state
                        |
                        v
              +------------------+
              |       QEMU       |
              |   VM1 VM2 VM3    |
              +------------------+

That does not mean every VM can already survive every kernel upgrade.

The state of a hypervisor consists of more than RAM. KVM, IOMMU, interrupts and, depending on the configuration, devices assigned through VFIO may all be involved.

Each subsystem must know how to preserve and restore its part of the state correctly.

That is precisely why LUO is being designed as extensible infrastructure rather than as a simple live-kernel-upgrade command.

Google provides luo-agent to manage the process

Alongside the components entering the Linux kernel, Google maintains the open-source luo-agent project under the GPL-2.0 license. It provides userspace components for controlling Live Update Orchestrator.

The project mainly consists of two tools:

luod
luoctl

luod is the daemon responsible for controlling /dev/liveupdate, maintaining sessions and coordinating clients.

luoctl provides the administrative interface.

Building the current project from source uses Meson and Ninja. On Debian or Ubuntu, the dependencies listed by the project can be installed with:

sudo apt-get install meson ninja-build libjson-c-devCode language: JavaScript (javascript)

The project can then be built and installed:

meson setup build
ninja -C build
sudo ninja -C build install

The daemon is designed to operate as a systemd service:

sudo systemctl enable --now luod

From there, the workflow starts to look familiar to a Linux administrator.

A new kernel can be loaded in advance:

sudo luoctl load /boot/vmlinuz \
  --initrd /boot/initrd.img \
  --reuse-cmdline

Registered clients can be inspected with:

sudo luoctl listCode language: PHP (php)

And the transition can eventually be triggered with:

sudo luoctl kexec

This sequence should not be interpreted as a recipe to test directly on a production host. The current luo-agent project requires Linux 6.19 or later with CONFIG_LIVEUPDATE enabled, along with appropriate support in the workloads that need to survive the transition.

The interesting part of luod: applications preserve their own state

The architecture chosen for luo-agent follows a fairly Unix-like approach.

luod owns /dev/liveupdate, but it does not try to understand the internal workings of every application.

Clients connect through a Unix domain socket:

/run/luod/liveupdate.sock

and subscribe using an identifier.

A simplified version of the protocol looks like this:

SUBSCRIBE
    |
    v
PRESERVATION_REQUEST
    |
    v
START_PRESERVATION + Session FD
    |
    v
Client preserves resources
    |
    v
READY
    |
    v
kexec
    |
    v
CLAIM
    |
    v
Recover Session FD

File descriptors can be passed between processes using SCM_RIGHTS.

This architecture means luod does not need detailed knowledge of QEMU, a database engine or a container runtime.

Each client knows what it needs to preserve.

The daemon coordinates when it happens.

systemd 261 also enters the picture

Another development of particular interest to Linux administrators is support in systemd 261.

systemd already has a file descriptor store that allows services to hand descriptors to the service manager for preservation.

Directives such as:

FileDescriptorStoreMax=

and:

FileDescriptorStorePreserve=

control this behaviour.

With LUO, the possibility emerges of preserving those descriptors across certain kexec transitions.

That creates interesting scenarios beyond virtual machines.

Consider a cache containing tens of gigabytes of data in memory through memfd. Restarting the service and completely rebuilding that cache can take considerable time.

If the descriptor and its associated memory can survive the kernel transition, the new process could reclaim the resource instead of reconstructing it.

Similar approaches could eventually become relevant to container runtimes, databases and network services.

There is still an important distinction between what the architecture makes possible and what is currently ready for generic production use.

What about sockets and network connections?

This is where the problem becomes considerably more difficult.

Keeping RAM alive is relatively straightforward compared with preserving the complete state of an active Linux system.

A real application may depend on:

memory;
open files;
sockets;
pipes;
devices;
network interfaces;
TCP connections;
VFIO;
IOMMU;
internal driver state.

Keeping a TCP connection alive while one kernel disappears and another takes over requires preserving far more information than the file descriptor representing the socket.

LUO will therefore become more useful as more kernel subsystems gain the ability to serialize, preserve and restore their state.

This is also where CRIU, or Checkpoint/Restore in Userspace, becomes relevant.

CRIU can freeze supported processes, save their state and restore them later. It is particularly well known in the container world and for process migration.

KHO, LUO and CRIU are not direct alternatives.

They operate at different layers and could eventually complement one another:

KHO
 |
 +-- preserves memory across kernels

LUO
 |
 +-- coordinates resources and applications

CRIU
 |
 +-- checkpoints and restores processes

Combining these mechanisms could allow Linux to tackle live kernel transitions from several layers of the operating system.

Live patching and LUO are likely to coexist

There is also little reason for administrators to replace kpatch, Ksplice or similar technologies with LUO automatically.

If a vulnerability can be addressed with a well-tested live patch, applying that patch may be considerably less invasive than replacing the entire kernel.

The problem comes when the system eventually needs the complete updated kernel.

Live patches can accumulate, and not every kernel modification is suitable for live patching.

LUO potentially provides another route: eventually running the fully updated kernel without necessarily evacuating every workload from the host first.

Both approaches could therefore coexist in future infrastructure.

An operator might deploy a live patch immediately after a critical vulnerability is disclosed. Later, during a maintenance window, the host could transition to the fully patched kernel using LUO with much less disruption than a conventional reboot.

Is LUO ready to test on Ubuntu?

Testing this technology and deploying it in production are two very different decisions.

KHO and LUO already have components in the upstream Linux kernel, and systemd is gaining related support, but the infrastructure is still evolving.

A kernel command line for a system specifically prepared for experimentation could include:

GRUB_CMDLINE_LINUX="kho=on kho_scratch=16M,512M,256M liveupdate=on"Code language: JavaScript (javascript)

GRUB would then need to be updated according to the distribution, followed by an initial reboot to activate the parameters.

But enabling these options does not magically provide transparent kernel upgrades.

An administrator should first check the running kernel:

uname -r

Then inspect its configuration:

grep -E 'CONFIG_KEXEC_HANDOVER|CONFIG_LIVEUPDATE' \
  /boot/config-$(uname -r)Code language: JavaScript (javascript)

And verify whether the live-update interface is available:

ls -l /dev/liveupdate

Applications also need to be explicitly prepared to participate in the process.

For that reason, LUO is currently most interesting for labs, kernel developers, Linux distribution engineers, cloud providers and teams operating large virtualization platforms.

It is not yet a drop-in replacement for the classic:

sudo reboot

on an ordinary Linux server.

Why this could matter to Linux administrators

The most interesting part of LUO is not being able to claim that Linux “never needs to reboot.”

Linux will continue to require conventional reboots for many reasons. Firmware updates, hardware changes, certain drivers and physical problems do not disappear because kexec exists.

The more important idea is separating two operations that administrators have historically treated as closely linked:

change the kernel
        =
interrupt host workloads

KHO and LUO are trying to weaken that dependency.

On a laptop, the difference may not matter very much. A 30-second reboot is rarely a serious problem.

On a hypervisor running 80 virtual machines, it is a very different story.

And for a cloud provider operating tens of thousands of physical servers, upgrading kernels without first migrating every VM could materially change the operational cost of keeping infrastructure patched.

That is why KHO and LUO are worth watching even though they are not yet everyday tools in the sysadmin toolbox.

The really interesting milestone will come when an administrator can inspect an available kernel, load it, perform a controlled transition and find that the virtual machines, caches or other supported services running before the change are still there afterwards.

That would be a much bigger change than simply making reboot a few seconds faster.

Frequently Asked Questions

Does LUO allow Linux to update without rebooting?

LUO still performs a kernel transition using kexec. The difference is that it attempts to preserve resources and workloads across that transition, avoiding a conventional full reboot and reducing disruption for compatible services.

What is the difference between KHO and LUO?

KHO provides the low-level mechanism for preserving memory across a kexec transition. LUO adds infrastructure that allows userspace and kernel subsystems to register, preserve and recover the resources required by workloads.

Can LUO already be used on production servers?

Parts of the technology are already in the upstream kernel, but the infrastructure is still evolving and requires specific support from the resources and applications being preserved. For now, it should be approached primarily as technology for testing and carefully controlled deployments.

Will LUO replace Ksplice, kpatch or other live-patching technologies?

Not necessarily. Live patching modifies a kernel that remains running, while LUO is intended to enable a transition to another kernel. The two approaches can be complementary.

Sources: