Hi all,
I am encountering intermittent kernel paging issues on multiple nodes
These crashes happen very unpredictably—sometimes once a year, but other times daily. Here are some of the error logs I’ve been seeing:
Common modules in the stack traces include qemu-kvm, vhost, and kvm_amd, and the crash seems to consistently happen during KVM operations. I’ve also observed the issue across different BIOS versions and hardware.
Has anyone experienced similar issues or have suggestions on how to tackle this? Any help would be greatly appreciated!
Thanks in advance.
[13412539.974665] kernel tried to execute NX-protected page - exploit attempt? (uid: 107)
[13412539.974705] BUG: unable to handle kernel paging request at ffff8a3f15b84790
[13412539.974729] PGD 766401067 P4D 766401067 PUD 6008dc063 PMD bd04e2063 PTE 80000016d5b84163
[13412539.974760] Oops: 0011 [#1] SMP NOPTI
[13412539.974777] CPU: 5 PID: 3648176 Comm: CPU 0/KVM Kdump: loaded Tainted: G W --------- - - 4.18.0-513.18.1.el8_9.x86_64 #1
[13412539.974814] Hardware name: To Be Filled By O.E.M. B650D4U-2L2T/BCM/B650D4U-2L2T/BCM, BIOS 4.09 10/02/2023
[13412539.974843] RIP: 0010:0xffff8a3f15b84790
[13412539.974859] Code: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 01 00 00 00 00 00 00 <00> 10 a2 43 90 b6 ff ff 00 00 00 00 00 00 00 00 c8 43 63 7b 29 8a
[13412539.974911] RSP: 0018:ffffb69019b17d38 EFLAGS: 00014012
[13412539.974929] RAX: 0000000000000000 RBX: 0000000000000001 RCX: ffff8a298281f438
[13412539.974953] RDX: 0000000000000007 RSI: 0000000080000001 RDI: ffff8a3f15b858b8
[13412539.974977] RBP: ffff8a3f15b858b8 R08: 0000000000000001 R09: 0000000000000001
[13412539.974999] R10: 0000000000000000 R11: 0000000000000001 R12: ffff8a3f15b86048
[13412539.975005] R13: ffff8a3f15b85820 R14: 0000000000000001 R15: ffff8a386d95b800
[13412539.975005] FS: 00007f352656d700(0000) GS:ffff8a47e8140000(0000) knlGS:0000000000000000
[13412539.975005] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[13412539.975005] CR2: ffff8a3f15b84790 CR3: 00000004f5eb0000 CR4: 0000000000750ee0
[13412539.975005] PKRU: 55555554
[13412539.975005] Call Trace:
[13412539.975005] ? __die_body+0x1a/0x60
[13412539.975005] ? no_context+0x1ba/0x3f0
[13412539.975005] ? __bad_area_nosemaphore+0x16c/0x1c0
[13412539.975005] ? spurious_kernel_fault+0x1ed/0x250
[13412539.975005] ? do_page_fault+0x37/0x12d
[13412539.975005] ? page_fault+0x1e/0x30
[13412539.975005] ? kvm_skip_emulated_instruction+0x3d/0x60 [kvm]
[13412539.975005] ? complete_fast_pio_in+0x75/0xc0 [kvm]
[13412539.975005] ? kvm_arch_vcpu_ioctl_run+0x592/0x600 [kvm]
[13412539.975005] ? kvm_vcpu_ioctl+0x2c9/0x640 [kvm]
[13412539.975005] ? srso_alias_return_thunk+0x5/0xfcdfd
[13412539.975005] ? do_vfs_ioctl+0xa4/0x690
[13412539.975005] ? srso_alias_return_thunk+0x5/0xfcdfd
[13412539.975005] ? syscall_trace_enter+0x1ff/0x2d0
[13412539.975005] ? ksys_ioctl+0x64/0xa0
[13412539.975005] ? __x64_sys_ioctl+0x16/0x20
[13412539.975005] ? do_syscall_64+0x5b/0x1b0
[13412539.975005] ? entry_SYSCALL_64_after_hwframe+0x61/0xc6
[13412539.975005] Modules linked in: loop ebt_ip6 dm_snapshot dm_bufio ebt_arp ebt_ip nft_compat nft_counter nf_tables libcrc32c vhost_net vhost vhost_iotlb tap tun nfnetlink bridge stp llc fuse sunrpc snd_sof_amd_rembrandt ipmi_ssif snd_sof_amd_renoir snd_hda_codec_hdmi snd_sof_amd_acp snd_sof_pci intel_rapl_msr snd_sof_xtensa_dsp snd_sof intel_rapl_common snd_sof_utils snd_hda_intel edac_mce_amd snd_intel_dspcfg snd_intel_sdw_acpi snd_soc_core snd_hda_codec snd_compress kvm_amd snd_hda_core snd_pci_acp6x snd_hwdep vfat dm_mod fat kvm snd_seq snd_seq_device snd_pcm irqbypass crct10dif_pclmul crc32_pclmul snd_timer snd_pci_acp5x ghash_clmulni_intel snd_rn_pci_acp3x cdc_ether sp5100_tco acpi_ipmi wmi_bmof snd usbnet snd_acp_config rapl pcspkr i2c_piix4 joydev ccp soundcore snd_soc_acpi mii wmi ipmi_si ipmi_devintf ipmi_msghandler gpio_amdpt gpio_generic acpi_cpufreq amdgpu ext4 mbcache jbd2 drm_ttm_helper ttm raid1 iommu_v2 gpu_sched drm_buddy drm_display_helper ast drm_shmem_helper
[13412539.975005] drm_kms_helper syscopyarea sysfillrect sysimgblt ahci drm libahci crc32c_intel bnxt_en libata igb nvme dca nvme_core i2c_algo_bit t10_pi video
[13412539.977078] CR2: ffff8a3f15b84790
[13412539.977078] WARNING: CPU: 5 PID: 3648176 at arch/x86/kernel/traps.c:958 do_debug+0x2a5/0x350
[13412539.977078] Modules linked in: loop ebt_ip6 dm_snapshot dm_bufio ebt_arp ebt_ip nft_compat nft_counter nf_tables libcrc32c vhost_net vhost vhost_iotlb tap tun nfnetlink bridge stp llc fuse sunrpc snd_sof_amd_rembrandt ipmi_ssif snd_sof_amd_renoir snd_hda_codec_hdmi snd_sof_amd_acp snd_sof_pci intel_rapl_msr snd_sof_xtensa_dsp snd_sof intel_rapl_common snd_sof_utils snd_hda_intel edac_mce_amd snd_intel_dspcfg snd_intel_sdw_acpi snd_soc_core snd_hda_codec snd_compress kvm_amd snd_hda_core snd_pci_acp6x snd_hwdep vfat dm_mod fat kvm snd_seq snd_seq_device snd_pcm irqbypass crct10dif_pclmul crc32_pclmul snd_timer snd_pci_acp5x ghash_clmulni_intel snd_rn_pci_acp3x cdc_ether sp5100_tco acpi_ipmi wmi_bmof snd usbnet snd_acp_config rapl pcspkr i2c_piix4 joydev ccp soundcore snd_soc_acpi mii wmi ipmi_si ipmi_devintf ipmi_msghandler gpio_amdpt gpio_generic acpi_cpufreq amdgpu ext4 mbcache jbd2 drm_ttm_helper ttm raid1 iommu_v2 gpu_sched drm_buddy drm_display_helper ast drm_shmem_helper
[71216.532124] Hardware name: To Be Filled By O.E.M. B650D4U-2L2T/BCM/B650D4U-2L2T/BCM, BIOS 2.09 03/14/2023
[71216.532100] CPU: 7 PID: 3476684 Comm: qemu-kvm Kdump: loaded Not tainted 4.18.0-513.18.1.el8_9.x86_64 #1
[71216.532085] Oops: 0010 [#1] SMP NOPTI
[71216.532063] PGD 197fc13067 P4D 197fc13067 PUD 197fc14063 PMD 0
[71216.532026] BUG: unable to handle kernel paging request at ffffffff8b1806ec
[71216.532157] RIP: 0010:0xffffffff8b1806ec
[2887077.636866] Hardware name: To Be Filled By O.E.M. B650D4U-2L2T/BCM/B650D4U-2L2T/BCM, BIOS 3.17 05/24/2023
[2887077.636714] CPU: 17 PID: 2328072 Comm: qemu-kvm Kdump: loaded Not tainted 4.18.0-513.18.1.el8_9.x86_64 #1
[2887077.636653] Oops: 0011 [#1] SMP NOPTI
[2887077.636541] PGD 574e13067 P4D 574e13067 PUD 574e14063 PMD 80000005748001e1
[2887077.636429] BUG: unable to handle kernel paging request at ffffffffaa259010
[2887077.637019] RIP: 0010:__start_BTF+0x12ba8/0x40acf0
[2887077.688478] entry_SYSCALL_64_after_hwframe+0x61/0xc6
[2887077.686423] do_syscall_64+0x5b/0x1b0
[2887077.684279] __x64_sys_ppoll+0xbf/0x120
[2887077.682054] ? compat_poll_select_copy_remaining+0x150/0x150
[2887077.679725] ? compat_poll_select_copy_remaining+0x150/0x150
[2887077.690490] RIP: 0033:0x7f917c937bb6
[4441534.178219] Hardware name: To Be Filled By O.E.M. B650D4U-2L2T/BCM/B650D4U-2L2T/BCM, BIOS 3.17 05/24/2023
[4441534.178060] CPU: 13 PID: 1855530 Comm: vhost-1855509 Kdump: loaded Not tainted 4.18.0-513.18.1.el8_9.x86_64 #1
[4441534.178000] Oops: 0002 [#1] SMP NOPTI
[4441534.177958] PGD 0 P4D 0
[4441534.177854] BUG: unable to handle kernel paging request at 00000000ffffffff
[4441534.178372] RIP: 0010:__vhost_add_used_n+0x16/0x220 [vhost]