Kernel 4.18.0-372.19.1.el8_6.x86_64 panics system after update

Support
1

Running AlmaLinux 8.6 on Linode and just did an update that resulted in a panic upon reboot.

To update, I did the following:

  1. Update:
$ yum update
Last metadata expiration check: 1:29:17 ago on Sat Aug  6 22:48:21 2022.
Dependencies resolved.
===========================================================================================================================
 Package                            Architecture         Version                                Repository            Size
===========================================================================================================================
Installing:
 kernel                             x86_64               4.18.0-372.19.1.el8_6                  baseos               8.0 M
 kernel-core                        x86_64               4.18.0-372.19.1.el8_6                  baseos                39 M
 kernel-debug-devel                 x86_64               4.18.0-372.19.1.el8_6                  baseos                21 M
 kernel-devel                       x86_64               4.18.0-372.19.1.el8_6                  baseos                21 M
 kernel-modules                     x86_64               4.18.0-372.19.1.el8_6                  baseos                32 M
 kernel-modules-extra               x86_64               4.18.0-372.19.1.el8_6                  baseos               8.7 M
Upgrading:
 kernel-headers                     x86_64               4.18.0-372.19.1.el8_6                  baseos               9.4 M
 kernel-tools                       x86_64               4.18.0-372.19.1.el8_6                  baseos               8.3 M
 kernel-tools-libs                  x86_64               4.18.0-372.19.1.el8_6                  baseos               8.1 M

Transaction Summary
===========================================================================================================================
Install  6 Packages
Upgrade  3 Packages

Total download size: 156 M
Is this ok [y/N]: y
Downloading Packages:
(1/9): kernel-4.18.0-372.19.1.el8_6.x86_64.rpm                                              27 MB/s | 8.0 MB     00:00
(2/9): kernel-debug-devel-4.18.0-372.19.1.el8_6.x86_64.rpm                                  38 MB/s |  21 MB     00:00
(3/9): kernel-devel-4.18.0-372.19.1.el8_6.x86_64.rpm                                        49 MB/s |  21 MB     00:00
(4/9): kernel-core-4.18.0-372.19.1.el8_6.x86_64.rpm                                         41 MB/s |  39 MB     00:00
(5/9): kernel-modules-extra-4.18.0-372.19.1.el8_6.x86_64.rpm                                29 MB/s | 8.7 MB     00:00
(6/9): kernel-headers-4.18.0-372.19.1.el8_6.x86_64.rpm                                      21 MB/s | 9.4 MB     00:00
(7/9): kernel-tools-4.18.0-372.19.1.el8_6.x86_64.rpm                                        19 MB/s | 8.3 MB     00:00
(8/9): kernel-modules-4.18.0-372.19.1.el8_6.x86_64.rpm                                      32 MB/s |  32 MB     00:01
(9/9): kernel-tools-libs-4.18.0-372.19.1.el8_6.x86_64.rpm                                   42 MB/s | 8.1 MB     00:00
---------------------------------------------------------------------------------------------------------------------------
Total                                                                                       77 MB/s | 156 MB     00:02
Running transaction check
Transaction check succeeded.
Running transaction test
Transaction test succeeded.
Running transaction
  Preparing        :                                                                                                   1/1
  Running scriptlet: kernel-core-4.18.0-372.19.1.el8_6.x86_64                                                          1/1
  Installing       : kernel-core-4.18.0-372.19.1.el8_6.x86_64                                                         1/12
  Running scriptlet: kernel-core-4.18.0-372.19.1.el8_6.x86_64                                                         1/12
  Installing       : kernel-modules-4.18.0-372.19.1.el8_6.x86_64                                                      2/12
  Running scriptlet: kernel-modules-4.18.0-372.19.1.el8_6.x86_64                                                      2/12
  Upgrading        : kernel-tools-libs-4.18.0-372.19.1.el8_6.x86_64                                                   3/12
  Running scriptlet: kernel-tools-libs-4.18.0-372.19.1.el8_6.x86_64                                                   3/12
  Upgrading        : kernel-tools-4.18.0-372.19.1.el8_6.x86_64                                                        4/12
  Installing       : kernel-4.18.0-372.19.1.el8_6.x86_64                                                              5/12
  Installing       : kernel-modules-extra-4.18.0-372.19.1.el8_6.x86_64                                                6/12
  Running scriptlet: kernel-modules-extra-4.18.0-372.19.1.el8_6.x86_64                                                6/12
  Upgrading        : kernel-headers-4.18.0-372.19.1.el8_6.x86_64                                                      7/12
  Installing       : kernel-devel-4.18.0-372.19.1.el8_6.x86_64                                                        8/12
  Running scriptlet: kernel-devel-4.18.0-372.19.1.el8_6.x86_64                                                        8/12
  Installing       : kernel-debug-devel-4.18.0-372.19.1.el8_6.x86_64                                                  9/12
  Running scriptlet: kernel-debug-devel-4.18.0-372.19.1.el8_6.x86_64                                                  9/12
  Cleanup          : kernel-headers-4.18.0-372.9.1.el8.x86_64                                                        10/12
  Cleanup          : kernel-tools-4.18.0-372.9.1.el8.x86_64                                                          11/12
  Cleanup          : kernel-tools-libs-4.18.0-372.9.1.el8.x86_64                                                     12/12
  Running scriptlet: kernel-tools-libs-4.18.0-372.9.1.el8.x86_64                                                     12/12
  Running scriptlet: kernel-core-4.18.0-372.19.1.el8_6.x86_64                                                        12/12
  Running scriptlet: kernel-tools-libs-4.18.0-372.9.1.el8.x86_64                                                     12/12
  Verifying        : kernel-4.18.0-372.19.1.el8_6.x86_64                                                              1/12
  Verifying        : kernel-core-4.18.0-372.19.1.el8_6.x86_64                                                         2/12
  Verifying        : kernel-debug-devel-4.18.0-372.19.1.el8_6.x86_64                                                  3/12
  Verifying        : kernel-devel-4.18.0-372.19.1.el8_6.x86_64                                                        4/12
  Verifying        : kernel-modules-4.18.0-372.19.1.el8_6.x86_64                                                      5/12
  Verifying        : kernel-modules-extra-4.18.0-372.19.1.el8_6.x86_64                                                6/12
  Verifying        : kernel-headers-4.18.0-372.19.1.el8_6.x86_64                                                      7/12
  Verifying        : kernel-headers-4.18.0-372.9.1.el8.x86_64                                                         8/12
  Verifying        : kernel-tools-4.18.0-372.19.1.el8_6.x86_64                                                        9/12
  Verifying        : kernel-tools-4.18.0-372.9.1.el8.x86_64                                                          10/12
  Verifying        : kernel-tools-libs-4.18.0-372.19.1.el8_6.x86_64                                                  11/12
  Verifying        : kernel-tools-libs-4.18.0-372.9.1.el8.x86_64                                                     12/12

Upgraded:
  kernel-headers-4.18.0-372.19.1.el8_6.x86_64                    kernel-tools-4.18.0-372.19.1.el8_6.x86_64
  kernel-tools-libs-4.18.0-372.19.1.el8_6.x86_64
Installed:
  kernel-4.18.0-372.19.1.el8_6.x86_64                         kernel-core-4.18.0-372.19.1.el8_6.x86_64
  kernel-debug-devel-4.18.0-372.19.1.el8_6.x86_64             kernel-devel-4.18.0-372.19.1.el8_6.x86_64
  kernel-modules-4.18.0-372.19.1.el8_6.x86_64                 kernel-modules-extra-4.18.0-372.19.1.el8_6.x86_64

Complete!
  1. Update GRUB 2:
$ grub2-mkconfig -o /boot/grub2/grub.cfg
Generating grub configuration file ...
Found linux image: /boot/vmlinuz-4.18.0-372.19.1.el8_6.x86_64
Found linux image: /boot/vmlinuz-4.18.0-372.9.1.el8.x86_64
Found initrd image: /boot/initramfs-4.18.0-372.9.1.el8.x86_64.img
Found linux image: /boot/vmlinuz-0-rescue-4f09fa5fdd3642fa85221d7c11370603
Found initrd image: /boot/initramfs-0-rescue-4f09fa5fdd3642fa85221d7c11370603.img
done
  1. Reboot ($ reboot) to /boot/vmlinuz-4.18.0-372.19.1.el8_6.x86_64 and see panic:
...
[    0.638194] evm: Initialising EVM extended attributes:
[    0.639435] evm: security.selinux
[    0.640246] evm: security.ima
[    0.640973] evm: security.capability
[    0.641800] evm: HMAC attrs: 0x1
[    0.643096] rtc_cmos 00:00: setting system clock to 2022-08-07 04:21:22 UTC )
[    0.645303] md: Waiting for all devices to be available before autodetect
[    0.646718] md: If you don't use raid, use raid=noautodetect
[    0.648396] md: Autodetecting RAID arrays.
[    0.649358] md: autorun ...
[    0.650033] md: ... autorun DONE.
[    0.650885] List of all partitions:
[    0.651756] No filesystem could mount root, tried:
[    0.651758]
[    0.653332] Kernel panic - not syncing: VFS: Unable to mount root fs on unkn)
[    0.655152] CPU: 3 PID: 1 Comm: swapper/0 Not tainted 4.18.0-372.19.1.el8_6.1
[    0.656845] Hardware name: QEMU Standard PC (Q35 + ICH9, 2009), BIOS rel-1.14
[    0.659136] Call Trace:
[    0.659754]  dump_stack+0x41/0x60
[    0.660532]  panic+0xe7/0x2ac
[    0.661292]  mount_block_root+0x2be/0x2e2
[    0.662252]  ? do_early_param+0x91/0x91
[    0.663080]  prepare_namespace+0x135/0x16b
[    0.663971]  kernel_init_freeable+0x203/0x22d
[    0.664917]  ? rest_init+0xaa/0xaa
[    0.665731]  kernel_init+0xa/0xfb
[    0.666510]  ret_from_fork+0x35/0x40
[    0.667415] Kernel Offset: 0x1ce00000 from 0xffffffff81000000 (relocation ra)
[    0.669546] ---[ end Kernel panic - not syncing: VFS: Unable to mount root f-

It seems fairly evident that the problem is that it’s not finding the root volume, however the previous kernel (/boot/vmlinuz-4.18.0-372.9.1.el8.x86_64) still boots fine if I select that instead. With an appropriate timeout I can pick the working kernel or I can issue this in GRUB for successful boot:

set root=(hd0)
linux /boot/vmlinuz-4.18.0-372.9.1.el8.x86_64 root=/dev/sda ro crashkernel=auto rhgb console=ttyS0,19200n8 net.ifnames=0
initrd /boot/initramfs-4.18.0-372.9.1.el8.x86_64.img
boot

I found the numbering kind of odd insomuch that the one booting is ...9.1... and the one failing is ...19.1... - perhaps that’s to be expected, but odd in mind that they’re only 10 off:

  • Success: /boot/vmlinuz-4.18.0-372.9.1.el8.x86_64
  • Failure: /boot/vmlinuz-4.18.0-372.19.1.el8_6.x86_64

In the /boot/ folder I see the following kernels and initrds present:

-rwxr-xr-x.  1 root root  10M May 10 09:04 vmlinuz-4.18.0-372.9.1.el8.x86_64
-rw-------.  1 root root  25M Jun 16 19:19 initramfs-4.18.0-372.9.1.el8.x86_64.img
-rwxr-xr-x.  1 root root  10M Aug  2 13:50 vmlinuz-4.18.0-372.19.1.el8_6.x86_64

The only glaring thing to me is that .19. seems missing the initramfs.

Any ideas about what may be going on here and how to resolve this panic situation with the mysterious missing root for .19.?

2

Sounds as if something went wrong during the kernel update. The easiest way to fix this will be to reinstall the latest kernel.

3

Can you please elaborate? I’ve basically installed it twice at this point with the same result each time. I suspect a missing the missing initramfs for the .19. kernel to be the/a problem, but of course I’m not certain.

Are you thinking yum reinstall kernel would remedy this?

4

I have seen cases in which initramfs was missing. In those cases yum reinstall the kernel usually takes care of the problem. Perhaps you may want to uninstall the latest kernel and then install it fresh?

5

I guess I’ll take this thread a little farther since I want to really make sure I understand and get around the corners of this bumpy ride. If anyone could chime into whether or not I’m missing anything, it would be appreciated.

I still cannot get seemingly standard methods (ie, installing from the distributions package) to install the initramfs img file.

The thing that bothers me is that the package that’s being installed SHOULD have the /boot/initramfs-4.18.0-372.19.1.el8_6.x86_64.img file in it: https://almalinux.pkgs.org/8/almalinux-baseos-x86_64/kernel-core-4.18.0-372.19.1.el8_6.x86_64.rpm.html

Even after running dnf reinstall kernel-core I do not see /boot/initramfs-4.18.0-372.19.1.el8_6.x86_64.img, note:

$ dnf reinstall kernel-core
Last metadata expiration check: 2:43:05 ago on Sun Aug  7 16:53:07 2022.
Dependencies resolved.
===========================================================================================================================
 Package                      Architecture            Version                                Repository               Size
===========================================================================================================================
Reinstalling:
 kernel-core                  x86_64                  4.18.0-372.9.1.el8                     baseos                   39 M
 kernel-core                  x86_64                  4.18.0-372.19.1.el8_6                  baseos                   39 M

Transaction Summary
===========================================================================================================================

Total download size: 79 M
Installed size: 137 M
Is this ok [y/N]: y
Downloading Packages:
(1/2): kernel-core-4.18.0-372.19.1.el8_6.x86_64.rpm                                        6.1 MB/s |  39 MB     00:06
(2/2): kernel-core-4.18.0-372.9.1.el8.x86_64.rpm                                           5.9 MB/s |  39 MB     00:06
---------------------------------------------------------------------------------------------------------------------------
Total                                                                                       12 MB/s |  79 MB     00:06
Running transaction check
Transaction check succeeded.
Running transaction test
Transaction test succeeded.
Running transaction
  Preparing        :                                                                                                   1/1
  Reinstalling     : kernel-core-4.18.0-372.9.1.el8.x86_64                                                             1/4
  Running scriptlet: kernel-core-4.18.0-372.9.1.el8.x86_64                                                             1/4
  Reinstalling     : kernel-core-4.18.0-372.19.1.el8_6.x86_64                                                          2/4
  Running scriptlet: kernel-core-4.18.0-372.19.1.el8_6.x86_64                                                          2/4
  Running scriptlet: kernel-core-4.18.0-372.9.1.el8.x86_64                                                             3/4
  Cleanup          : kernel-core-4.18.0-372.9.1.el8.x86_64                                                             3/4
  Running scriptlet: kernel-core-4.18.0-372.19.1.el8_6.x86_64                                                          4/4
  Cleanup          : kernel-core-4.18.0-372.19.1.el8_6.x86_64                                                          4/4
  Running scriptlet: kernel-core-4.18.0-372.9.1.el8.x86_64                                                             4/4
  Running scriptlet: kernel-core-4.18.0-372.19.1.el8_6.x86_64                                                          4/4
  Verifying        : kernel-core-4.18.0-372.19.1.el8_6.x86_64                                                          1/4
  Verifying        : kernel-core-4.18.0-372.19.1.el8_6.x86_64                                                          2/4
  Verifying        : kernel-core-4.18.0-372.9.1.el8.x86_64                                                             3/4
  Verifying        : kernel-core-4.18.0-372.9.1.el8.x86_64                                                             4/4

Reinstalled:
  kernel-core-4.18.0-372.9.1.el8.x86_64                      kernel-core-4.18.0-372.19.1.el8_6.x86_64

Complete!

And take a peek:

$ ls -trlah /boot/initramfs-4.18.0-372.9.1.el8.x86_64*
-rw-------. 1 root root 25M Jun 16 19:19 /boot/initramfs-4.18.0-372.9.1.el8.x86_64.img
-rw-------. 1 root root 28M Aug  7 01:10 /boot/initramfs-4.18.0-372.9.1.el8.x86_64kdump.img

Apparent that it’s not there.

So to be more thorough:

$ dnf reinstall kernel-4.18.0-372.19.1.el8_6.x86_64 kernel-modules-4.18.0-372.19.1.el8_6.x86_64 kernel-devel-4.18.0-372.19.1.el8_6.x86_64 kernel-tools-4.18.0-372.19.1.el8_6.x86_64 kernel-headers-4.18.0-372.19.1.el8_6.x86_64 python3-perf-4.18.0-372.19.1.el8_6.x86_64 kernel-modules-extra-4.18.0-372.19.1.el8_6.x86_64 kernel-debug-devel-4.18.0-372.19.1.el8_6.x86_64 kernel-tools-libs-4.18.0-372.19.1.el8_6.x86_64 kernel-core-4.18.0-372.19.1.el8_6.x86_64
Last metadata expiration check: 2:45:47 ago on Sun Aug  7 16:53:07 2022.
Dependencies resolved.
===========================================================================================================================
 Package                            Architecture         Version                                Repository            Size
===========================================================================================================================
Reinstalling:
 kernel                             x86_64               4.18.0-372.19.1.el8_6                  baseos               8.0 M
 kernel-core                        x86_64               4.18.0-372.19.1.el8_6                  baseos                39 M
 kernel-debug-devel                 x86_64               4.18.0-372.19.1.el8_6                  baseos                21 M
 kernel-devel                       x86_64               4.18.0-372.19.1.el8_6                  baseos                21 M
 kernel-headers                     x86_64               4.18.0-372.19.1.el8_6                  baseos               9.4 M
 kernel-modules                     x86_64               4.18.0-372.19.1.el8_6                  baseos                32 M
 kernel-modules-extra               x86_64               4.18.0-372.19.1.el8_6                  baseos               8.7 M
 kernel-tools                       x86_64               4.18.0-372.19.1.el8_6                  baseos               8.3 M
 kernel-tools-libs                  x86_64               4.18.0-372.19.1.el8_6                  baseos               8.1 M
 python3-perf                       x86_64               4.18.0-372.19.1.el8_6                  baseos               8.2 M

Transaction Summary
===========================================================================================================================

Total download size: 164 M
Installed size: 203 M
Is this ok [y/N]: y
Downloading Packages:
(1/10): kernel-4.18.0-372.19.1.el8_6.x86_64.rpm                                            3.9 MB/s | 8.0 MB     00:02
(2/10): kernel-debug-devel-4.18.0-372.19.1.el8_6.x86_64.rpm                                5.6 MB/s |  21 MB     00:03
(3/10): kernel-headers-4.18.0-372.19.1.el8_6.x86_64.rpm                                    6.0 MB/s | 9.4 MB     00:01
(4/10): kernel-devel-4.18.0-372.19.1.el8_6.x86_64.rpm                                      4.5 MB/s |  21 MB     00:04
(5/10): kernel-core-4.18.0-372.19.1.el8_6.x86_64.rpm                                       5.6 MB/s |  39 MB     00:07
(6/10): kernel-modules-extra-4.18.0-372.19.1.el8_6.x86_64.rpm                              6.1 MB/s | 8.7 MB     00:01
(7/10): kernel-tools-4.18.0-372.19.1.el8_6.x86_64.rpm                                      4.9 MB/s | 8.3 MB     00:01
(8/10): kernel-tools-libs-4.18.0-372.19.1.el8_6.x86_64.rpm                                 7.2 MB/s | 8.1 MB     00:01
(9/10): python3-perf-4.18.0-372.19.1.el8_6.x86_64.rpm                                      6.4 MB/s | 8.2 MB     00:01
(10/10): kernel-modules-4.18.0-372.19.1.el8_6.x86_64.rpm                                   6.7 MB/s |  32 MB     00:04
---------------------------------------------------------------------------------------------------------------------------
Total                                                                                       16 MB/s | 164 MB     00:10
Running transaction check
Transaction check succeeded.
Running transaction test
Transaction test succeeded.
Running transaction
  Preparing        :                                                                                                   1/1
  Running scriptlet: kernel-core-4.18.0-372.19.1.el8_6.x86_64                                                          1/1
  Reinstalling     : kernel-core-4.18.0-372.19.1.el8_6.x86_64                                                         1/20
  Running scriptlet: kernel-core-4.18.0-372.19.1.el8_6.x86_64                                                         1/20
  Reinstalling     : kernel-modules-4.18.0-372.19.1.el8_6.x86_64                                                      2/20
  Running scriptlet: kernel-modules-4.18.0-372.19.1.el8_6.x86_64                                                      2/20
  Reinstalling     : kernel-tools-libs-4.18.0-372.19.1.el8_6.x86_64                                                   3/20
  Running scriptlet: kernel-tools-libs-4.18.0-372.19.1.el8_6.x86_64                                                   3/20
  Reinstalling     : kernel-tools-4.18.0-372.19.1.el8_6.x86_64                                                        4/20
  Reinstalling     : kernel-4.18.0-372.19.1.el8_6.x86_64                                                              5/20
  Reinstalling     : kernel-modules-extra-4.18.0-372.19.1.el8_6.x86_64                                                6/20
  Running scriptlet: kernel-modules-extra-4.18.0-372.19.1.el8_6.x86_64                                                6/20
  Reinstalling     : python3-perf-4.18.0-372.19.1.el8_6.x86_64                                                        7/20
  Reinstalling     : kernel-headers-4.18.0-372.19.1.el8_6.x86_64                                                      8/20
  Reinstalling     : kernel-devel-4.18.0-372.19.1.el8_6.x86_64                                                        9/20
  Running scriptlet: kernel-devel-4.18.0-372.19.1.el8_6.x86_64                                                        9/20
  Reinstalling     : kernel-debug-devel-4.18.0-372.19.1.el8_6.x86_64                                                 10/20
  Running scriptlet: kernel-debug-devel-4.18.0-372.19.1.el8_6.x86_64                                                 10/20
  Cleanup          : kernel-4.18.0-372.19.1.el8_6.x86_64                                                             11/20
  Running scriptlet: kernel-4.18.0-372.19.1.el8_6.x86_64                                                             11/20
  Cleanup          : kernel-headers-4.18.0-372.19.1.el8_6.x86_64                                                     12/20
  Cleanup          : kernel-devel-4.18.0-372.19.1.el8_6.x86_64                                                       13/20
  Cleanup          : kernel-debug-devel-4.18.0-372.19.1.el8_6.x86_64                                                 14/20
  Cleanup          : kernel-tools-4.18.0-372.19.1.el8_6.x86_64                                                       15/20
  Cleanup          : kernel-modules-extra-4.18.0-372.19.1.el8_6.x86_64                                               16/20
  Running scriptlet: kernel-modules-extra-4.18.0-372.19.1.el8_6.x86_64                                               16/20
  Cleanup          : kernel-modules-4.18.0-372.19.1.el8_6.x86_64                                                     17/20
  Running scriptlet: kernel-modules-4.18.0-372.19.1.el8_6.x86_64                                                     17/20
  Running scriptlet: kernel-core-4.18.0-372.19.1.el8_6.x86_64                                                        18/20
  Cleanup          : kernel-core-4.18.0-372.19.1.el8_6.x86_64                                                        18/20
  Cleanup          : kernel-tools-libs-4.18.0-372.19.1.el8_6.x86_64                                                  19/20
  Running scriptlet: kernel-tools-libs-4.18.0-372.19.1.el8_6.x86_64                                                  19/20
  Cleanup          : python3-perf-4.18.0-372.19.1.el8_6.x86_64                                                       20/20
  Running scriptlet: kernel-core-4.18.0-372.19.1.el8_6.x86_64                                                        20/20
  Running scriptlet: python3-perf-4.18.0-372.19.1.el8_6.x86_64                                                       20/20
  Verifying        : kernel-4.18.0-372.19.1.el8_6.x86_64                                                              1/20
  Verifying        : kernel-4.18.0-372.19.1.el8_6.x86_64                                                              2/20
  Verifying        : kernel-core-4.18.0-372.19.1.el8_6.x86_64                                                         3/20
  Verifying        : kernel-core-4.18.0-372.19.1.el8_6.x86_64                                                         4/20
  Verifying        : kernel-debug-devel-4.18.0-372.19.1.el8_6.x86_64                                                  5/20
  Verifying        : kernel-debug-devel-4.18.0-372.19.1.el8_6.x86_64                                                  6/20
  Verifying        : kernel-devel-4.18.0-372.19.1.el8_6.x86_64                                                        7/20
  Verifying        : kernel-devel-4.18.0-372.19.1.el8_6.x86_64                                                        8/20
  Verifying        : kernel-headers-4.18.0-372.19.1.el8_6.x86_64                                                      9/20
  Verifying        : kernel-headers-4.18.0-372.19.1.el8_6.x86_64                                                     10/20
  Verifying        : kernel-modules-4.18.0-372.19.1.el8_6.x86_64                                                     11/20
  Verifying        : kernel-modules-4.18.0-372.19.1.el8_6.x86_64                                                     12/20
  Verifying        : kernel-modules-extra-4.18.0-372.19.1.el8_6.x86_64                                               13/20
  Verifying        : kernel-modules-extra-4.18.0-372.19.1.el8_6.x86_64                                               14/20
  Verifying        : kernel-tools-4.18.0-372.19.1.el8_6.x86_64                                                       15/20
  Verifying        : kernel-tools-4.18.0-372.19.1.el8_6.x86_64                                                       16/20
  Verifying        : kernel-tools-libs-4.18.0-372.19.1.el8_6.x86_64                                                  17/20
  Verifying        : kernel-tools-libs-4.18.0-372.19.1.el8_6.x86_64                                                  18/20
  Verifying        : python3-perf-4.18.0-372.19.1.el8_6.x86_64                                                       19/20
  Verifying        : python3-perf-4.18.0-372.19.1.el8_6.x86_64                                                       20/20

Reinstalled:
  kernel-4.18.0-372.19.1.el8_6.x86_64                             kernel-core-4.18.0-372.19.1.el8_6.x86_64
  kernel-debug-devel-4.18.0-372.19.1.el8_6.x86_64                 kernel-devel-4.18.0-372.19.1.el8_6.x86_64
  kernel-headers-4.18.0-372.19.1.el8_6.x86_64                     kernel-modules-4.18.0-372.19.1.el8_6.x86_64
  kernel-modules-extra-4.18.0-372.19.1.el8_6.x86_64               kernel-tools-4.18.0-372.19.1.el8_6.x86_64
  kernel-tools-libs-4.18.0-372.19.1.el8_6.x86_64                  python3-perf-4.18.0-372.19.1.el8_6.x86_64

Complete!

So look again, still not there:

$ ls -trlah /boot/*372.19*
-rw-------. 1 root root 4.2M Aug  2 13:50 /boot/System.map-4.18.0-372.19.1.el8_6.x86_64
-rw-r--r--. 1 root root 192K Aug  2 13:50 /boot/config-4.18.0-372.19.1.el8_6.x86_64
-rwxr-xr-x. 1 root root  10M Aug  2 13:50 /boot/vmlinuz-4.18.0-372.19.1.el8_6.x86_64
lrwxrwxrwx. 1 root root   52 Aug  7 19:41 /boot/symvers-4.18.0-372.19.1.el8_6.x86_64.gz -> /lib/modules/4.18.0-372.19.1.el8_6.x86_64/symvers.gz

Triple sanity check to see what should be going on:

$ dnf repoquery -l kernel-core-4.18.0-372.19.1.el8_6.x86_64 | grep initramfs
Last metadata expiration check: 2:59:44 ago on Sun Aug  7 16:53:07 2022.
/boot/initramfs-4.18.0-372.19.1.el8_6.x86_64.img
$ dnf repoquery --file /boot/initramfs-4.18.0-372.19.1.el8_6.x86_64.img
Last metadata expiration check: 3:01:15 ago on Sun Aug  7 16:53:07 2022.
kernel-core-0:4.18.0-372.19.1.el8_6.x86_64
$ rpm -ql kernel-core-4.18.0-372.19.1.el8_6.x86_64 | grep initramfs
/boot/initramfs-4.18.0-372.19.1.el8_6.x86_64.img
$ rpm -qf /boot/initramfs-4.18.0-372.19.1.el8_6.x86_64.img
kernel-core-4.18.0-372.19.1.el8_6.x86_64

So it should be there, but it is not… :tired_face:

As I am still using root for these operations, I thought I’d go spelunking throughout the entire drive, but still no luck:

$ find / | grep initramfs-4.18.0-372.19.1.el8_6.x86_64.img
find: '/proc/180568': No such file or directory
$

What is going on here???

I see NO errors or indications from the installations / reinstallations that there is something wrong or the initramfs file is having problems…

I’m really not sure how to handle this situation, so I have attempted further to do the following to generate this mysterious missing file myself:

$ dracut /boot/initramfs-4.18.0-372.19.1.el8_6.x86_64.img 4.18.0-372.19.1.el8_6.x86_64
$ ls -trlah /boot/*372.19*
-rw-------. 1 root root 4.2M Aug  2 13:50 /boot/System.map-4.18.0-372.19.1.el8_6.x86_64
-rw-r--r--. 1 root root 192K Aug  2 13:50 /boot/config-4.18.0-372.19.1.el8_6.x86_64
-rwxr-xr-x. 1 root root  10M Aug  2 13:50 /boot/vmlinuz-4.18.0-372.19.1.el8_6.x86_64
lrwxrwxrwx. 1 root root   52 Aug  7 19:41 /boot/symvers-4.18.0-372.19.1.el8_6.x86_64.gz -> /lib/modules/4.18.0-372.19.1.el8_6.x86_64/symvers.gz
-rw-------. 1 root root  23M Aug  7 20:02 /boot/initramfs-4.18.0-372.19.1.el8_6.x86_64.img
$ grub2-mkconfig -o /boot/grub2/grub.cfg
Generating grub configuration file ...
Found linux image: /boot/vmlinuz-4.18.0-372.19.1.el8_6.x86_64
Found initrd image: /boot/initramfs-4.18.0-372.19.1.el8_6.x86_64.img
Found linux image: /boot/vmlinuz-4.18.0-372.9.1.el8.x86_64
Found initrd image: /boot/initramfs-4.18.0-372.9.1.el8.x86_64.img
done

So as you can tell, I was able to generate an appropriate initramfs image for /boot/vmlinuz-4.18.0-372.19.1.el8_6.x86_64 it seems, but is this really the correct approach to solve this?

Isn’t something broken with AlmaLinux kernel packaging for this to occur?

6

Can you try this? Uninstall both ‘kernel’ and ‘kernel-core’ for 4.18.0-372.19.1. And then install the kernel.

7

A little nervous about doing this, getting the following @toracat:

$ dnf remove kernel-core-4.18.0-372.19.1.el8_6.x86_64 kernel-core-4.18.0-372.19.1.el8_6.x86_64
Error:
 Problem: The operation would result in removing the following protected packages: kernel-core
(try to add '--skip-broken' to skip uninstallable packages or '--nobest' to use not only best candidate packages)
#
# So I thought I'd expand the removal:
#
$ dnf remove kernel-4.18.0-372.19.1.el8_6.x86_64 kernel-modules-4.18.0-372.19.1.el8_6.x86_64 kernel-devel-4.18.0-372.19.1.el8_6.x86_64 kernel-tools-4.18.0-372.19.1.el8_6.x86_64 kernel-headers-4.18.0-372.19.1.el8_6.x86_64 python3-perf-4.18.0-372.19.1.el8_6.x86_64 kernel-modules-extra-4.18.0-372.19.1.el8_6.x86_64 kernel-debug-devel-4.18.0-372.19.1.el8_6.x86_64 kernel-tools-libs-4.18.0-372.19.1.el8_6.x86_64 kernel-core-4.18.0-372.19.1.el8_6.x86_64
Error:
 Problem: The operation would result in removing the following protected packages: kernel-core
(try to add '--skip-broken' to skip uninstallable packages or '--nobest' to use not only best candidate packages)
8

In /etc/yum.conf, what is installonly_limit= set to ? The default is 3.

9 
$ cat /etc/yum.conf
[main]
exclude=bind-chroot dovecot* exim* filesystem nsd* p0f php* proftpd* pure-ftpd*
tolerant=1
errorlevel=1
gpgcheck=1
installonly_limit=3
clean_requirements_on_remove=True
best=True
skip_if_unavailable=False
10

I wonder about this option …

clean_requirements_on_remove=True
11

I’m not sure what to say / do here. For what it’s worth, I’m also running cPanel on this server and so I’m just a little bit leery of doing anything that might create 20-30 minutes worth of downtime at the moment with a snapshot-restore cycle on the Linode if things go south and this week is very hectic in terms of investing some evening time.

I guess I just wish there were more clarity on this situation. Are any kernel package maintainers floating about here that might have more context and experience in debugging what might be going on?

I’m thinking that there should be some mechanism to trace out the yum reinstall process more fully to see where and how the installation of the img file is failing…

12

That is the default on dnf of EL8, according to man-page.

Remove dependencies that are no longer used during dnf remove. A package only qualifies for removal via clean_requirements_on_remove if it was installed through DNF but not on explicit user request, i.e. it was pulled in as a dependency. The default is True.

The protect_running_kernel would explain the removal issue, if one would try to remove 4.18.0-372.19.1 while running 4.18.0-372.19.1, but since one cannot boot that kernel here, that should not be the case.

Is it possible that the cPanel has made changes that do produce the error(s)?