Updating your kernel when /boot is too small
Some cloud images provide a /boot filesystem which is too small to hold the
required files to boot 2 different kernel versions.
In this tip we'll see how to identify when an update was unsuccesfull and what to do in order to run the latest kernel in the next reboot.
Identifying /boot size
This boot filesystem is only 495 MB in size, and size it's a RHEL9 system, it is too small to hold two kernels:
# df -h /boot/ Filesystem Size Used Avail Use% Mounted on /dev/vda3 495M 360M 136M 73% /boot
Identifying issues while upgradng
When we upgrade the system with yum, the following error is shown on screen:
# yum upgrade -y [...] Running scriptlet: kernel-core-5.14.0-427.22.1.el9_4.x86_64 963/963 cp: error writing '/boot/initramfs-5.14.0-427.22.1.el9_4.x86_64.img': No space left on device dracut: dracut: creation of /boot/initramfs-5.14.0-427.22.1.el9_4.x86_64.img failed warning: %posttrans(kernel-core-5.14.0-427.22.1.el9_4.x86_64) scriptlet failed, exit status 1 Error in POSTTRANS scriptlet in rpm package kernel-core
This means that the newly-regenerated initramfs is incomplete/corrupt as it didn't have enough space to be properly created. The next boot using that initramfs will fail and leave you with an unbootable kernel.
We can also see errors in the messages file:
# cat /var/log/messages| grep -P "dracut: creation.*failed" Jul 2 10:11:42 server.example.com dracut[27623]: dracut: creation of /boot/initramfs-5.14.0-427.22.1.el9_4.x86_64.img fail
Luckily the old kernel is still working and we can just boot the old kernel to get the system properly updated.
Erasing the old kernel and rebuilding the initramfs
We can now identify the old and new versions of the kernel:
# rpm -qa| grep kern | sort
kernel-5.14.0-362.13.1.el9_3.x86_64
kernel-5.14.0-427.22.1.el9_4.x86_64
kernel-core-5.14.0-362.13.1.el9_3.x86_64
kernel-core-5.14.0-427.22.1.el9_4.x86_64
kernel-modules-5.14.0-362.13.1.el9_3.x86_64
kernel-modules-5.14.0-427.22.1.el9_4.x86_64
kernel-modules-core-5.14.0-362.13.1.el9_3.x86_64
kernel-modules-core-5.14.0-427.22.1.el9_4.x86_64
We have 5.14.0-362.13.1.el9_3 (old) and 5.14.0-427.22.1.el9_4 (new).
We can remove the old version of the kernel (the running one).
BEWARE THIS IS A DANGEROUS STEP! You may render your system unbootable if you don't follow the steps properly and manage to fully install the new kernel.
Remove the old kernel packages with rpm . dnf will not let you do so because
it's the running kernel.
# rpm -qa| grep kern | grep 5.14.0-362.13.1.el9_3 | xargs rpm -e
/usr/sbin/weak-modules: line 1086: cd: /lib/modules/5.14.0-362.13.1.el9_3.x86_64/weak-updates: No such file or directory
warning: file /lib/modules/5.14.0-362.13.1.el9_3.x86_64/modules.builtin.modinfo: remove failed: No such file or directory
warning: file /lib/modules/5.14.0-362.13.1.el9_3.x86_64/modules.builtin: remove failed: No such file or directory
Once you've removed the old kernel, you should have plenty of space in /boot
to let it install the new kernel and regenerate the new initramfs automatically
with dracut.
A simple way to do that is to just reinstall the kernel package with dnf:
# yum reinstall kernel && reboot Updating Subscription Management repositories. Last metadata expiration check: 0:24:06 ago on Tue Jul 2 09:55:07 2024. Dependencies resolved. ===================================================================================================== Package Architecture Version Repository Size ===================================================================================================== Reinstalling: kernel x86_64 5.14.0-427.22.1.el9_4 rhel-9-for-x86_64-baseos-rpms 5.5 M Transaction Summary ===================================================================================================== Total download size: 5.5 M Installed size: 0 Is this ok [y/N]: y
This task ensures that:
- The kernel package is reinstalled.
- The initramfs is regenerated for all installed kernel packages (only one in our current scenario).
Once this step is done, you'll be running the latest kernel.
Happy hacking!
