Some cloud images provide a /boot filesystem which is too small to hold the required files to boot 2 different kernel versions.

In this tip we'll see how to identify when an update was unsuccesfull and what to do in order to run the latest kernel in the next reboot.

Identifying /boot size

This boot filesystem is only 495 MB in size, and size it's a RHEL9 system, it is too small to hold two kernels:

# df -h /boot/
Filesystem      Size  Used Avail Use% Mounted on
/dev/vda3       495M  360M  136M  73% /boot

Identifying issues while upgradng

When we upgrade the system with yum, the following error is shown on screen:

# yum upgrade -y
[...]
  Running scriptlet: kernel-core-5.14.0-427.22.1.el9_4.x86_64                                                                                                                                                                  963/963 
cp: error writing '/boot/initramfs-5.14.0-427.22.1.el9_4.x86_64.img': No space left on device
dracut: dracut: creation of /boot/initramfs-5.14.0-427.22.1.el9_4.x86_64.img failed
warning: %posttrans(kernel-core-5.14.0-427.22.1.el9_4.x86_64) scriptlet failed, exit status 1

Error in POSTTRANS scriptlet in rpm package kernel-core

This means that the newly-regenerated initramfs is incomplete/corrupt as it didn't have enough space to be properly created. The next boot using that initramfs will fail and leave you with an unbootable kernel.

We can also see errors in the messages file:

# cat /var/log/messages| grep -P "dracut: creation.*failed" 
Jul  2 10:11:42 server.example.com dracut[27623]: dracut: creation of /boot/initramfs-5.14.0-427.22.1.el9_4.x86_64.img fail

Luckily the old kernel is still working and we can just boot the old kernel to get the system properly updated.

Erasing the old kernel and rebuilding the initramfs

We can now identify the old and new versions of the kernel:

# rpm -qa| grep kern | sort
kernel-5.14.0-362.13.1.el9_3.x86_64
kernel-5.14.0-427.22.1.el9_4.x86_64
kernel-core-5.14.0-362.13.1.el9_3.x86_64
kernel-core-5.14.0-427.22.1.el9_4.x86_64
kernel-modules-5.14.0-362.13.1.el9_3.x86_64
kernel-modules-5.14.0-427.22.1.el9_4.x86_64
kernel-modules-core-5.14.0-362.13.1.el9_3.x86_64
kernel-modules-core-5.14.0-427.22.1.el9_4.x86_64

We have 5.14.0-362.13.1.el9_3 (old) and 5.14.0-427.22.1.el9_4 (new).

We can remove the old version of the kernel (the running one).

BEWARE THIS IS A DANGEROUS STEP! You may render your system unbootable if you don't follow the steps properly and manage to fully install the new kernel.

Remove the old kernel packages with rpm . dnf will not let you do so because it's the running kernel.

# rpm -qa| grep kern | grep 5.14.0-362.13.1.el9_3 | xargs rpm -e 
/usr/sbin/weak-modules: line 1086: cd: /lib/modules/5.14.0-362.13.1.el9_3.x86_64/weak-updates: No such file or directory
warning: file /lib/modules/5.14.0-362.13.1.el9_3.x86_64/modules.builtin.modinfo: remove failed: No such file or directory
warning: file /lib/modules/5.14.0-362.13.1.el9_3.x86_64/modules.builtin: remove failed: No such file or directory

Once you've removed the old kernel, you should have plenty of space in /boot to let it install the new kernel and regenerate the new initramfs automatically with dracut.

A simple way to do that is to just reinstall the kernel package with dnf:

# yum reinstall kernel && reboot
Updating Subscription Management repositories.
Last metadata expiration check: 0:24:06 ago on Tue Jul  2 09:55:07 2024.
Dependencies resolved.
=====================================================================================================
 Package               Architecture    Version                 Repository                     Size
=====================================================================================================
Reinstalling:
 kernel                x86_64          5.14.0-427.22.1.el9_4   rhel-9-for-x86_64-baseos-rpms  5.5 M

Transaction Summary
=====================================================================================================
Total download size: 5.5 M
Installed size: 0  
Is this ok [y/N]: y

This task ensures that:

The kernel package is reinstalled.
The initramfs is regenerated for all installed kernel packages (only one in our current scenario).

Once this step is done, you'll be running the latest kernel.

Happy hacking!