6.12.0-55.21.1.el10_0.x86_64_v2 kernel panic

Support
1

After updating to 6.12.0-55.21.1.el10_0.x86_64_v2 on AlmaLinux release 10.0
I have a kernel panic with unable to mount root fs or unknown block.

kernel-panic-10
kernel-panic-10727×406 6.94 KB

I’m running on _v2 with root on LVM.

Previous kernel 6.12.0-55.9.1.el10_0.x86_64_v2 works fine.

I’ve tried to recreate initramfs with dracut as well as re-install kernel and didn’t notice any errors there.

Jul 28 13:42:47 root[5191]: os-prober: debug: running /usr/libexec/os-probes/50mounted-tests on /dev/vda1
Jul 28 13:42:47 root[5191]: 50mounted-tests: debug: /dev/vda1 type not recognised; skipping
Jul 28 13:42:47 root[5191]: os-prober: debug: running /usr/libexec/os-probes/mounted/05efi on mounted /dev/vda2
Jul 28 13:42:47 root[5191]: 05efi: debug: Not on UEFI platform
Jul 28 13:42:47 root[5191]: os-prober: debug: running /usr/libexec/os-probes/mounted/10freedos on mounted /dev/vda2
Jul 28 13:42:47 root[5191]: 10freedos: debug: /dev/vda2 is not a FAT partition: exiting
Jul 28 13:42:47 root[5191]: os-prober: debug: running /usr/libexec/os-probes/mounted/10qnx on mounted /dev/vda2
Jul 28 13:42:47 root[5191]: 10qnx: debug: /dev/vda2 is not a QNX4 partition: exiting
Jul 28 13:42:47 root[5191]: os-prober: debug: running /usr/libexec/os-probes/mounted/20macosx on mounted /dev/vda2
Jul 28 13:42:47 macosx-prober[5248]: debug: /dev/vda2 is not an HFS+ partition: exiting
Jul 28 13:42:47 root[5191]: os-prober: debug: running /usr/libexec/os-probes/mounted/20microsoft on mounted /dev/vda2
Jul 28 13:42:47 root[5191]: 20microsoft: debug: /dev/vda2 is not a MS partition: exiting
Jul 28 13:42:47 root[5191]: os-prober: debug: running /usr/libexec/os-probes/mounted/30utility on mounted /dev/vda2
Jul 28 13:42:47 root[5191]: 30utility: debug: /dev/vda2 is not a FAT partition: exiting
Jul 28 13:42:47 root[5191]: os-prober: debug: running /usr/libexec/os-probes/mounted/40lsb on mounted /dev/vda2
Jul 28 13:42:47 root[5191]: os-prober: debug: running /usr/libexec/os-probes/mounted/70hurd on mounted /dev/vda2
Jul 28 13:42:47 root[5191]: os-prober: debug: running /usr/libexec/os-probes/mounted/80minix on mounted /dev/vda2
Jul 28 13:42:47 root[5191]: os-prober: debug: running /usr/libexec/os-probes/mounted/83haiku on mounted /dev/vda2
Jul 28 13:42:47 root[5191]: 83haiku: debug: /dev/vda2 is not a BeFS partition: exiting
Jul 28 13:42:47 root[5191]: os-prober: debug: running /usr/libexec/os-probes/mounted/90linux-distro on mounted /dev/vda2
Jul 28 13:42:47 root[5191]: os-prober: debug: running /usr/libexec/os-probes/mounted/90solaris on mounted /dev/vda2
Jul 28 13:42:47 root[5191]: os-prober: debug: running /usr/libexec/os-probes/50mounted-tests on /dev/vda3
Jul 28 13:42:47 root[5191]: 50mounted-tests: debug: /dev/vda3 is an LVM member; skipping
Jul 28 13:42:47 root[5191]: os-prober: debug: /dev/mapper/vg0-swap: is active swap
Jul 28 13:42:52 dracut[5350]: dracut-105-4.el10_0
Jul 28 13:42:52 dracut[5353]: Executing: /usr/bin/dracut -f --kernel-image /lib/modules/6.12.0-55.21.1.el10_0.x86_64_v2/vmlinuz --kver 6.12.0-55.21.1.el10_0.x86_64_v2 /boot/initramfs-6.12.0-55.21.1.el10_0.x86_64_v2.img
Jul 28 13:42:53 dracut[5353]: 62bluetooth: Could not find any command of ‘/usr/lib/bluetooth/bluetoothd /usr/libexec/bluetooth/bluetoothd’!
Jul 28 13:42:53 dracut[5353]: 95nfs: Could not find any command of ‘rpcbind portmap’!
Jul 28 13:42:53 dracut[5353]: 62bluetooth: Could not find any command of ‘/usr/lib/bluetooth/bluetoothd /usr/libexec/bluetooth/bluetoothd’!
Jul 28 13:42:54 dracut[5353]: 95nfs: Could not find any command of ‘rpcbind portmap’!
Jul 28 13:42:54 dracut[5353]: *** Including module: bash ***
Jul 28 13:42:54 dracut[5353]: *** Including module: shell-interpreter ***
Jul 28 13:42:54 dracut[5353]: *** Including module: systemd ***
Jul 28 13:42:54 dracut[5353]: *** Including module: fips ***
Jul 28 13:42:55 dracut[5353]: *** Including module: fips-crypto-policies ***
Jul 28 13:42:55 dracut[5353]: *** Including module: systemd-ask-password ***
Jul 28 13:42:55 dracut[5353]: *** Including module: systemd-initrd ***
Jul 28 13:42:55 dracut[5353]: *** Including module: systemd-journald ***
Jul 28 13:42:55 dracut[5353]: *** Including module: systemd-modules-load ***
Jul 28 13:42:55 dracut[5353]: *** Including module: systemd-sysctl ***
Jul 28 13:42:55 dracut[5353]: *** Including module: systemd-sysusers ***
Jul 28 13:42:55 dracut[5353]: *** Including module: systemd-tmpfiles ***
Jul 28 13:42:55 dracut[5353]: *** Including module: systemd-udevd ***
Jul 28 13:42:55 dracut[5353]: *** Including module: i18n ***
Jul 28 13:42:55 dracut[5353]: *** Including module: prefixdevname ***
Jul 28 13:42:55 dracut[5353]: *** Including module: dm ***
Jul 28 13:42:55 dracut[5353]: *** Including module: kernel-modules ***
Jul 28 13:42:55 kernel: block dm-0: the capability attribute has been deprecated.
Jul 28 13:42:56 dracut[5353]: *** Including module: kernel-modules-extra ***
Jul 28 13:42:56 dracut[5353]: *** Including module: lvm ***
Jul 28 13:42:56 dracut[5353]: *** Including module: qemu ***
Jul 28 13:42:56 dracut[5353]: *** Including module: hwdb ***
Jul 28 13:42:56 dracut[5353]: *** Including module: resume ***
Jul 28 13:42:56 dracut[5353]: *** Including module: rootfs-block ***
Jul 28 13:42:56 dracut[5353]: *** Including module: terminfo ***
Jul 28 13:42:56 dracut[5353]: *** Including module: udev-rules ***
Jul 28 13:42:57 dracut[5353]: *** Including module: virtiofs ***
Jul 28 13:42:57 dracut[5353]: *** Including module: dracut-systemd ***
Jul 28 13:42:57 dracut[5353]: *** Including module: usrmount ***
Jul 28 13:42:57 dracut[5353]: *** Including module: base ***
Jul 28 13:42:57 dracut[5353]: *** Including module: fs-lib ***
Jul 28 13:42:57 dracut[5353]: *** Including module: microcode_ctl-fw_dir_override ***
Jul 28 13:42:57 dracut[5353]: microcode_ctl module: mangling fw_dir
Jul 28 13:42:57 dracut[5353]: microcode_ctl: reset fw_dir to “/lib/firmware/updates /lib/firmware”
Jul 28 13:42:57 dracut[5353]: microcode_ctl: processing data directory “/usr/share/microcode_ctl/ucode_with_caveats/intel”…
Jul 28 13:42:57 dracut[5353]: microcode_ctl: configuration “intel” is ignored
Jul 28 13:42:57 dracut[5353]: microcode_ctl: processing data directory “/usr/share/microcode_ctl/ucode_with_caveats/intel-06-4f-01”…
Jul 28 13:42:57 dracut[5353]: microcode_ctl: configuration “intel-06-4f-01” is ignored
Jul 28 13:42:57 dracut[5353]: microcode_ctl: final fw_dir: “/lib/firmware/updates /lib/firmware”
Jul 28 13:42:57 dracut[5353]: *** Including module: openssl ***
Jul 28 13:42:57 dracut[5353]: *** Including module: shutdown ***
Jul 28 13:42:57 dracut[5353]: *** Including modules done ***
Jul 28 13:42:57 dracut[5353]: *** Installing kernel module dependencies ***
Jul 28 13:42:58 dracut[5353]: *** Installing kernel module dependencies done ***
Jul 28 13:42:58 dracut[5353]: *** Resolving executable dependencies ***
Jul 28 13:42:59 dracut[5353]: *** Resolving executable dependencies done ***
Jul 28 13:42:59 dracut[5353]: *** Hardlinking files ***
Jul 28 13:42:59 dracut[5353]: *** Hardlinking files done ***
Jul 28 13:42:59 dracut[5353]: *** Generating early-microcode cpio image ***
Jul 28 13:42:59 dracut[5353]: *** Constructing GenuineIntel.bin ***
Jul 28 13:42:59 dracut[5353]: *** Store current command line parameters ***
Jul 28 13:42:59 dracut[5353]: *** Creating image file ‘/boot/initramfs-6.12.0-55.21.1.el10_0.x86_64_v2.img’ ***
Jul 28 13:42:59 dracut[5353]: Using auto-determined compression method ‘gzip’
Jul 28 13:43:23 dracut[5353]: *** Creating initramfs image file ‘/boot/initramfs-6.12.0-55.21.1.el10_0.x86_64_v2.img’ done ***

2

Hi @bilias!

dracut -fv --add lvm --add-drivers "dm-mod dm-snapshot dm-mirror" /boot/initramfs-6.12.0-55.21.1.el10_0.x86_64_v2.img 6.12.0-55.21.1.el10_0.x86_64_v2
lsinitrd /boot/initramfs-6.12.0-55.21.1.el10_0.x86_64_v2.img | grep lvm

Ensure that the boot line contains something like:
root=/dev/mapper/vg0-root rd.lvm.lv=vg0/root
or
root=UUID=...

All the best! :v:

3

@Fekthis

Both initramfs as well as grub cmd contain the appropriate info

/boot/grub2/grub.cfg:
menuentry ‘AlmaLinux (6.12.0-55.21.1.el10_0.x86_64_v2) 10.0 (Purple Lion)’ --class almalinux --class gnu-linux --class gnu --class os --unrestricted $menuentry_id_option ‘gnulinux-6.12.0-55.21.1.el10_0.x86_64_v2-advanced-b22c5773-f0a8-4672-a474-cae3b4508020’ {
load_video
insmod gzio
insmod part_gpt
insmod xfs
set root=‘hd0,gpt2’
if [ x$feature_platform_search_hint = xy ]; then
search --no-floppy --fs-uuid --set=root --hint=‘hd0,gpt2’ 7f1666cb-07b2-4c32-b051-49b3ed44ba50
else
search --no-floppy --fs-uuid --set=root 7f1666cb-07b2-4c32-b051-49b3ed44ba50
fi
echo ‘Loading Linux 6.12.0-55.21.1.el10_0.x86_64_v2 …’
linux /vmlinuz-6.12.0-55.21.1.el10_0.x86_64_v2 root=/dev/mapper/vg0-root ro resume=UUID=da531428-91ce-49f0-852d-5932a666da8d rd.lvm.lv=vg0/root rd.lvm.lv=vg0/swap selinux=0
}

cat /proc/cmdline

BOOT_IMAGE=/vmlinuz-6.12.0-55.9.1.el10_0.x86_64_v2 root=/dev/mapper/vg0-root ro resume=UUID=da531428-91ce-49f0-852d-5932a666da8d rd.lvm.lv=vg0/root rd.lvm.lv=vg0/swap selinux=0

lsinitrd /boot/initramfs-6.12.0-55.21.1.el10_0.x86_64_v2.img|grep lvm
lvm
drwxr-xr-x 2 root root 0 Mar 6 02:00 etc/lvm
-rw-r–r-- 1 root root 114756 Feb 4 02:00 etc/lvm/lvm.conf
-rw-r–r-- 1 root root 2476 Feb 4 02:00 etc/lvm/lvmlocal.conf
-rw-r–r-- 1 root root 1308 Oct 13 2024 etc/udev/rules.d/64-lvm.rules
-rw-r–r-- 1 root root 2182 Feb 4 02:00 usr/lib/udev/rules.d/11-dm-lvm.rules
-rwxr-xr-x 1 root root 2740912 Feb 4 02:00 usr/sbin/lvm
-rwxr-xr-x 1 root root 4856 Oct 13 2024 usr/sbin/lvm_scan
-rwxr-xr-x 1 root root 427 Oct 13 2024 var/lib/dracut/hooks/cmdline/30-parse-lvm.sh

4

I had:

# uname -r
6.12.0-55.20.1.el10_0.x86_64_v2

did update, and the machine rebooted to:

# uname -r
6.12.0-55.21.1.el10_0.x86_64_v2

Alas, this is baremetal and / is on regular partition, not LVM. That is, I can only say that the 21 is not all bad.

The annoying thing about boot messages is that by the time the “Kernel panic” line shows, the actual, first error message was already many lines earlier and the virtual console/framebuffer does not have buffer to browse back to it.

5

Yes, I cannot get the framebuffer back to see other info or error message.

6

The problem after was with that resume line
blkid /dev/vg0/swap
/dev/vg0/swap: UUID=“da531428-91ce-49f0-852d-5932a666da8d” TYPE=“swap”

Removing the resume device, fixed the boot problem which I had with:

6.12.0-55.21.1.el10_0.x86_64_v2
and
6.12.0-55.22.1.el10_0.x86_64_v2

current cmd that works:
GRUB_CMDLINE_LINUX=“rd.lvm.lv=vg0/root rd.lvm.lv=vg0/swap selinux=0”

7

I had the same problem today after updating to the newest kernel.

I’ve pinned the problem to:
The new grub2.cfg auto generated by the kernel update does not include an initrd line

If I manual run grub2-mkconfig then initrd line is written in the file.

8

I had:

# ls -l /boot/{initramfs,vmlin}*  /boot/loader/entries/* /boot/grub2/grub* /boot/efi/EFI/almalinux/* | sed "s/1c2.*eea/1c2..eea/"
-rwx------. 1 root root       122 Jul  8 03:00 /boot/efi/EFI/almalinux/BOOTX64.CSV
-rwx------. 1 root root       144 Jul 22 13:41 /boot/efi/EFI/almalinux/grub.cfg
-rwx------. 1 root root   4019048 May 14 03:00 /boot/efi/EFI/almalinux/grubx64.efi
-rwx------. 1 root root    860184 Jul  8 03:00 /boot/efi/EFI/almalinux/mmx64.efi
-rwx------. 1 root root    959208 Jul  8 03:00 /boot/efi/EFI/almalinux/shim.efi
-rwx------. 1 root root    958600 Jul  8 03:00 /boot/efi/EFI/almalinux/shimx64-almalinux.efi
-rwx------. 1 root root    959208 Jul  8 03:00 /boot/efi/EFI/almalinux/shimx64.efi
-rw-------. 1 root root      7243 Aug 25 12:33 /boot/grub2/grub.cfg
-rw-r--r--. 1 root root      1024 Aug 25 12:33 /boot/grub2/grubenv
-rw-------. 1 root root 157958596 Jul 22 13:40 /boot/initramfs-0-rescue-1c2..eea.img
-rw-------. 1 root root  25100902 Jul 28 14:12 /boot/initramfs-6.12.0-55.21.1.el10_0.x86_64_v2.img
-rw-------. 1 root root  43971072 Jul 28 14:15 /boot/initramfs-6.12.0-55.21.1.el10_0.x86_64_v2kdump.img
-rw-------. 1 root root  25092904 Aug  6 16:05 /boot/initramfs-6.12.0-55.24.1.el10_0.x86_64_v2.img
-rw-------. 1 root root  36078080 Aug 25 12:31 /boot/initramfs-6.12.0-55.24.1.el10_0.x86_64_v2kdump.img
-rw-------. 1 root root  25078556 Aug 25 12:33 /boot/initramfs-6.12.0-55.27.1.el10_0.x86_64_v2.img
-rw-------. 1 root root  36078080 Aug 25 12:38 /boot/initramfs-6.12.0-55.27.1.el10_0.x86_64_v2kdump.img
-rw-r--r--. 1 root root       470 Aug 25 12:31 /boot/loader/entries/1c2..eea-0-rescue.conf
-rw-r--r--. 1 root root       458 Jul 28 14:12 /boot/loader/entries/1c2..eea-6.12.0-55.21.1.el10_0.x86_64_v2.conf
-rw-r--r--. 1 root root       458 Aug  6 16:05 /boot/loader/entries/1c2..eea-6.12.0-55.24.1.el10_0.x86_64_v2.conf
-rw-r--r--. 1 root root       458 Aug 25 12:33 /boot/loader/entries/1c2..eea-6.12.0-55.27.1.el10_0.x86_64_v2.conf
-rwxr-xr-x. 1 root root  15885696 Jul 22 13:38 /boot/vmlinuz-0-rescue-1c2..eea
-rwxr-xr-x. 1 root root  15861120 Jul 21 03:00 /boot/vmlinuz-6.12.0-55.21.1.el10_0.x86_64_v2
-rwxr-xr-x. 1 root root  15861120 Aug  4 03:00 /boot/vmlinuz-6.12.0-55.24.1.el10_0.x86_64_v2
-rwxr-xr-x. 1 root root  15861120 Aug 19 03:00 /boot/vmlinuz-6.12.0-55.27.1.el10_0.x86_64_v2

After dnf up I have:

# ls -l /boot/{initramfs,vmlin}*  /boot/loader/entries/* /boot/grub2/grub* /boot/efi/EFI/almalinux/* | sed "s/1c2.*eea/1c2..eea/"
-rwx------. 1 root root       122 Jul  8 03:00 /boot/efi/EFI/almalinux/BOOTX64.CSV
-rwx------. 1 root root       144 Jul 22 13:41 /boot/efi/EFI/almalinux/grub.cfg
-rwx------. 1 root root   4019048 May 14 03:00 /boot/efi/EFI/almalinux/grubx64.efi
-rwx------. 1 root root    860184 Jul  8 03:00 /boot/efi/EFI/almalinux/mmx64.efi
-rwx------. 1 root root    959208 Jul  8 03:00 /boot/efi/EFI/almalinux/shim.efi
-rwx------. 1 root root    958600 Jul  8 03:00 /boot/efi/EFI/almalinux/shimx64-almalinux.efi
-rwx------. 1 root root    959208 Jul  8 03:00 /boot/efi/EFI/almalinux/shimx64.efi
-rw-------. 1 root root      7243 Sep 22 17:30 /boot/grub2/grub.cfg
-rw-r--r--. 1 root root      1024 Sep 22 17:30 /boot/grub2/grubenv
-rw-------. 1 root root 157958596 Jul 22 13:40 /boot/initramfs-0-rescue-1c2..eea.img
-rw-------. 1 root root  25092904 Aug  6 16:05 /boot/initramfs-6.12.0-55.24.1.el10_0.x86_64_v2.img
-rw-------. 1 root root  36078080 Aug 25 12:31 /boot/initramfs-6.12.0-55.24.1.el10_0.x86_64_v2kdump.img
-rw-------. 1 root root  25078556 Aug 25 12:33 /boot/initramfs-6.12.0-55.27.1.el10_0.x86_64_v2.img
-rw-------. 1 root root  36078080 Aug 25 12:38 /boot/initramfs-6.12.0-55.27.1.el10_0.x86_64_v2kdump.img
-rw-------. 1 root root  25151019 Sep 22 17:31 /boot/initramfs-6.12.0-55.32.1.el10_0.x86_64_v2.img
-rw-r--r--. 1 root root       470 Aug 25 12:31 /boot/loader/entries/1c2..eea-0-rescue.conf
-rw-r--r--. 1 root root       458 Aug  6 16:05 /boot/loader/entries/1c2..eea-6.12.0-55.24.1.el10_0.x86_64_v2.conf
-rw-r--r--. 1 root root       458 Aug 25 12:33 /boot/loader/entries/1c2..eea-6.12.0-55.27.1.el10_0.x86_64_v2.conf
-rw-r--r--. 1 root root       458 Sep 22 17:31 /boot/loader/entries/1c2..eea-6.12.0-55.32.1.el10_0.x86_64_v2.conf
-rwxr-xr-x. 1 root root  15885696 Jul 22 13:38 /boot/vmlinuz-0-rescue-1c2..eea
-rwxr-xr-x. 1 root root  15861120 Aug  4 03:00 /boot/vmlinuz-6.12.0-55.24.1.el10_0.x86_64_v2
-rwxr-xr-x. 1 root root  15861120 Aug 19 03:00 /boot/vmlinuz-6.12.0-55.27.1.el10_0.x86_64_v2
-rwxr-xr-x. 1 root root  15865216 Sep 16 03:00 /boot/vmlinuz-6.12.0-55.32.1.el10_0.x86_64_v2

The /boot/grub2/grub.cfg and /boot/grub2/grubenv were touched, but
my initrd line is not in there te begin with. It is in:

# cat /boot/loader/entries/1c2*eea-6.12.0-55.32.1.el10_0.x86_64_v2.conf
title AlmaLinux (6.12.0-55.32.1.el10_0.x86_64_v2) 10.0 (Purple Lion)
version 6.12.0-55.32.1.el10_0.x86_64_v2
linux /vmlinuz-6.12.0-55.32.1.el10_0.x86_64_v2
initrd /initramfs-6.12.0-55.32.1.el10_0.x86_64_v2.img $tuned_initrd
options root=UUID=33d2637c-0231-486c-877e-e2d8dd15645d ro resume=UUID=407602af-49c7-4664-bd60-f42e1aba6c1f rhgb quiet crashkernel=2G-64G:256M,64G-:512M $tuned_params
grub_users $grub_users
grub_arg --unrestricted
grub_class almalinux

and after boot:

# uname -r
6.12.0-55.32.1.el10_0.x86_64_v2

AFAIK, the kernel install does not use grub2-mkconfig – grubby creates new BLS entry. Have you disabled use of BLS?

9

Yes I have it disabled in /etc/default/grub
GRUB_ENABLE_BLSCFG=false

cause I can’t remember to add the option --update-bls-cmdline when I manuall run grub2-mkconfig

In most of my servers (not 10, but 8 and 9) I have BLS disabled and kernel update, updates grub.cfg.

In this setup, grub.cfg is being updated (new kernel entry inserted) but with initrd missing, so it seems like a bug.

From my understanding you also verified this, correct?

10

I have only one el10_0.x86_64_v2 system and it does use BLS.