[BUG] NVIDIA kmod-nvidia-open mismatch with Kernel 5.14.0-611.45.1+

HI;

[BUG REPORT] NVIDIA driver breaks after kernel update — AlmaLinux 9.7 Machine: sirius (DELL T340 / RTX 3080) Date: April 23, 2026


Environment

  • OS: AlmaLinux 9.7

  • Working kernel: 5.14.0-611.41.1.el9_7.x86_64

  • Broken kernels: 611.45.1 and 611.49.1 (removed)

  • NVIDIA driver: 595.58.03 via kmod-nvidia-open (no DKMS)

  • Display manager: SDDM 0.20.0-3.el9.1


Problem

After a standard dnf update, the system booted on the latest kernels (611.45.1 then 611.49.1). NVIDIA module failed to load on both — SDDM would not start, leaving the machine with no graphical interface.

Root cause: kmod-nvidia-open is a pre-compiled module tied to a specific kernel version. No DKMS = no automatic recompilation when the kernel changes. If no matching kmod-nvidia package is available in the repo for the new kernel at update time, the GPU driver silently breaks on next boot.


Workaround applied

bash

# Rolled back to last working kernel
grubby --set-default /boot/vmlinuz-5.14.0-611.41.1.el9_7.x86_64

# Locked kernel to prevent auto-update
dnf install python3-dnf-plugin-versionlock
dnf versionlock add kernel-0:5.14.0-611.41.1.el9_7.*
dnf versionlock add kernel-core-0:5.14.0-611.41.1.el9_7.*
dnf versionlock add kernel-modules-0:5.14.0-611.41.1.el9_7.*

# Removed broken kernels
dnf remove kernel-5.14.0-611.45.1.el9_7.x86_64
dnf remove kernel-5.14.0-611.49.1.el9_7.x86_64

System is now stable but kernel frozen — no security updates possible until NVIDIA kmod catches up.


Questions for the team

  1. Is a kmod-nvidia-open-595.58.03 package planned/available for kernels 611.45.1 or 611.49.1?

  2. Is there a roadmap to ship nvidia-driver-dkms in the official AlmaLinux NVIDIA repo to avoid this class of issue entirely?

  3. Should there be a dependency or conflict declared in the repo to prevent kernel from updating without a matching kmod-nvidia?

Best Regards
Henri

I answered the query you posted in other thread:

“Have you considered using ELRepo’s kmod packages? In general, they survive kernel updates. There is a kmod-nvidia-open package available for el9.”

Dear toracat ;

according to proposal almalinux say “ almalinux 9 10 now with native support for nvidia drivers "

The promises of ELRepo's KABI (kernel binary interface) sound great in theory, but when Red Hat releases a kernel with structural changes (as is the case with these 611.x versions), KABI is no longer sufficient, and the .ko module must be recompiled. In this case, it's not associated with the compilation of the new kernels!

No, that's not enough!

Theory vs. Practice: Between kernel .41 (which works) and .49 (which breaks), there have been changes that the current pre-compiled module can't handle.

Build Lag: Even with ELRepo, there's often a gap of a few days between the release of the kernel by AlmaLinux and the availability of the compatible kmod.

Loading Failure: I tested it, and the result was a black screen. Reality proves the repository's theory wrong.
Dear Toracat; 
The promises of ELRepo's KABI (kernel binary interface) sound great in theory, but when Red Hat releases a kernel with structural changes (as is the case with these 611.x versions), KABI is no longer sufficient, and the .ko module must be recompiled.

If they claim it "survives" updates, you can politely point out that:

Theory vs. Practice: Between kernel .41 (which works) and .49 (which breaks), there were changes that the current pre-compiled module can't handle.

Build Lag: Even with ELRepo, there's often a gap of a few days between AlmaLinux's kernel release and the availability of the compatible kmod.

Loading Failure: You tested it, and the result was a black screen. Reality proves the repository's theory wrong.
( Hopefully, this has raised your awareness of this persistent problem. Note that until recently, for convenience, I successfully used the direct driver provided by Nvidia (.run) up until the second-to-last version, which no longer worked with newer kernel versions! sic!! )
 Best regards   
Henri 

HI,
New tests, new solution; bad result !
Subject: nvidia-driver 595.58.03 on AlmaLinux 9.7 — kmod missing for kernels 611.42 / 611.45 / 611.49

System: AlmaLinux 9.7 (Moss Jungle Cat) · RTX 3080 · kernel 5.14.0-611.49

Background: Previously running NVIDIA 580.x via ELRepo without issues. Following the official AlmaLinux procedure (blog post 2026-03-09) to migrate to the AlmaLinux-NVIDIA repo.

Root causes found after extensive troubleshooting:

  1. ELRepo 580.x packages (kmod-nvidia-580, nvidia-x11-drv-580, nvidia-x11-drv-libs-580) were still installed alongside the new 595 packages, causing a kernel module / userspace library version mismatch.
  2. After full cleanup and clean reinstall from AlmaLinux-NVIDIA only, nvidia_modeset and nvidia_drm still refused to load with Unknown symbol nvidia_get_rm_ops (err -22).
  3. Root cause: kmod-nvidia-open-595.58.03-1.el9 is compiled for kernel 5.14.0-611.41 only. No kmod has been published for 611.42, 611.45, or 611.49 — all three kernels currently in the grub menu.

Verification:

modinfo nvidia | grep vermagic
vermagic: 5.14.0-611.41.1.el9_7.x86_64
dnf list available ‘kmod-nvidia*’ --showduplicates | grep -E “611.42|611.45|611.49”
(no output)

Workaround applied: Reinstalled kernel 611.41 (dnf install kernel-5.14.0-611.41.1.el9_7.x86_64), set it as default, applied dnf versionlock to prevent automatic upgrade. Everything works correctly on 611.41.

Request: Please publish kmod-nvidia-open-595.58.03 built for kernels 611.42, 611.45, and 611.49 so that systems receiving standard AlmaLinux kernel updates do not lose their graphical environment silently.

Best regards
Henri

additional info: [Henry@sirius Téléchargements]$
sudo dnf list available ‘kmod-nvidia*’ --showduplicates
Dernière vérification de l’expiration des métadonnées effectuée il y a 0:11:10 le ven. 24 avril 2026 14:13:55.
Paquets disponibles
kmod-nvidia-open.x86_64 590.48.01-1.el9 almalinux-nvidia
kmod-nvidia-open.x86_64 595.45.04-1.el9 almalinux-nvidia
kmod-nvidia-open.x86_64 595.58.03-1.el9 almalinux-nvidia

The AlmaLinux-NVIDIA repository releases new versions of the driver but doesn’t recompile for new kernels as they come out.

This confirms that the problem is systemic: any AlmaLinux 9.7 user who follows the official procedure and receives a standard kernel update will end up in the same situation as you—no graphic, no warning.
Regards !

The ‘kmod-nvidia-open*.rpm’ from AlmaLinux does have postinstall scriptlet that runs weak-modules. A scritpt that:

# weak-modules - determine which modules are kABI compatible with installed
#                kernels and set up the symlinks in /lib/*/weak-updates.

That is the same thing as what the ELRepo’s packages do.


In other words, both AlmaLinux and ELRepo do apparently build (and sign) kernel modules that can be used by more than one version of the kernel – as long as kABI remains compatible.

Generally, each point update of RHEL does definitely change kABI, but change of kABI in the (critical) updates for one point update have been rare.

Yes, both AlmaLinux and ELRepo do have to rebuild if kABI does change. They do, once they learn about the change.


An alternative is to not use these pre-built modules, but use a package from NVidia’s repository that depends on DKMS to create the kernel modules on your machine for your kernel.

1 Like

HI Jlehtone

[“jlehtone, post:6, topic:7217”] The ‘kmod-nvidia-open*.rpm’ from AlmaLinux has a post-installation scriptlet that runs the ‘weak-modules’. A script that:

Thanks for your help, Jlhetone.

The real underlying problem is that the kABI changes between AlmaLinux 9.7 kernel updates—which is supposed to be rare on RHEL, but apparently it happens. And when it happens:

Precompiled kmods (AlmaLinux-NVIDIA, ELRepo) → no longer load
DKMS (.run or other) → recompilation fails

The real question is :

“Did the kABI change between 611.41 and 611.42 and +? If so, why don’t the kernel errata mention this change, and what is the expected timeframe for republishing the NVIDIA kmods?”

For now, in the best the best possible position, but blocked on update kernel:

Kernel 611.41 locked :white_check_mark:
Driver 595.58.03 working (discard screen at certain times with kde) :white_check_mark:
Forum informed :white_check_mark:jone,
Regards
Henri

I’d just like to follow up on your findings with the AlmaLinux 9.7 kernels and the ELRepo kmod-nvidia[-open] packages.

Using the official RHEL 9.7 with the matching kernels (-611.41.1, -611.42.1, -611.49.1), the kABI has not been broken with our NVIDIA kmods. For all the kernels, I was able to boot and login to GNOME without issue and also run some typical applications (nvidia-smi, glxinfo, glxgears, firefox on Youtube, etc.).

To me this indicates that the AlmaLinux kernel may have sourced an incorrect patch that broke kABI compatibility with the RHEL kernel.

1 Like

Following up on my post.

I just compared the kernel source of 5.14.0-611.49.1.el9_7.x86_64 between RHEL and AlmaLinux. Aside from the expected driver re-enablement stuff, it all looks identical.

The next thing I would check is if you’re running out of space on /boot. That can cause the weak-modules linking to prematurely exit. How many kernels do you have installed?

Here are two machines:

$ uname -r
5.14.0-611.47.1.el9_7.x86_64
$ modinfo nvidia | grep signer
signer:         DKMS module signing key
$ nvidia-smi | grep NVIDIA
| NVIDIA-SMI 590.48.01        Driver Version: 590.48.01      CUDA Version: 13.1  |
|   0  NVIDIA GeForce RTX 3070        Off |   00000000:01:00.0  On |                  N/A |
$ sudo dnf -q list kmod-nvidia-open-dkms
Installed Packages
kmod-nvidia-open-dkms.noarch           3:590.48.01-1.el9        @cuda-rhel9-x86_64

# uname -r
5.14.0-611.49.1.el9_7.x86_64
# modinfo nvidia | grep signer
signer:         DKMS module signing key
# nvidia-smi | grep NVIDIA
| NVIDIA-SMI 580.126.20       Driver Version: 580.126.20     CUDA Version: 13.0  |
# dnf -q list kmod-nvidia\*
Installed Packages
kmod-nvidia-latest-dkms.x86_64         3:580.126.20-1.el9       @cuda-rhel9-x86_64

Both do have more recent kernel and kernel module built by DKMS.

It is possible though that the 595 series driver has novel “features”.

HI,

You are right tqhoang, and I have investigated in your direction which I thought was consistent with what I had observed in my previous attempts to correct the problem. Here is the solution and the process of correcting the nvidia kernel/driver problem (-611.41.1, -611.42.1, -611.49.1).

[SOLVED] NVIDIA 595.58.03 on AlmaLinux 9.7 — working solution using DKMS from official CUDA repo

System: AlmaLinux 9.7 · RTX 3080 · Secure Boot disabled

Root cause confirmed: AlmaLinux kernel updates broke kABI compatibility — kmod-nvidia-open was only compiled for kernel 611.41. Kernels 611.42, 611.45, and 611.49 had no compatible kmod published in the AlmaLinux-NVIDIA repo.

Working solution:

bash

# 1 — Remove conflicting kmods
sudo dnf remove kmod-nvidia-open nvidia-open-kmod
sudo dnf config-manager --set-disabled elrepo elrepo-kernel

# 2 — Add official NVIDIA CUDA repo
sudo dnf config-manager --add-repo \
    https://developer.download.nvidia.com/compute/cuda/repos/rhel9/x86_64/cuda-rhel9.repo

# 3 — Install DKMS kmod from CUDA repo
sudo dnf install kmod-nvidia-latest-dkms-595.58.03

# 4 — Reinstall userspace packages
sudo dnf install nvidia-driver nvidia-driver-cuda nvidia-settings xorg-x11-nvidia

# 5 — Install kernel-devel for all installed kernels
sudo dnf install kernel-devel-$(uname -r)

# 6 — Compile for all kernels
sudo dkms autoinstall
sudo dkms autoinstall -k 5.14.0-611.49.1.el9_7.x86_64

# 7 — Set latest kernel as default and reboot
sudo grubby --set-default /boot/vmlinuz-5.14.0-611.49.1.el9_7.x86_64
sudo systemctl set-default graphical.target
sudo reboot
nvidia-smi
Wed Apr 29 08:33:00 2026       
+-----------------------------------------------------------------------------------------+
| NVIDIA-SMI 595.58.03              Driver Version: 595.58.03      CUDA Version: 13.2     |
+-----------------------------------------+------------------------+----------------------+
| GPU  Name                 Persistence-M | Bus-Id          Disp.A | Volatile Uncorr. ECC |
| Fan  Temp   Perf          Pwr:Usage/Cap |           Memory-Usage | GPU-Util  Compute M. |
|                                         |                        |               MIG M. |
|=========================================+========================+======================|
|   0  NVIDIA GeForce RTX 3080        Off |   00000000:01:00.0  On |                  N/A |
|  0%   36C    P8             26W /  320W |    1354MiB /  10240MiB |      0%      Default |
|                                         |                        |                  N/A |
+-----------------------------------------+------------------------+----------------------+

+-----------------------------------------------------------------------------------------+
| Processes:                                                                              |
|  GPU   GI   CI              PID   Type   Process name                        GPU Memory |
|        ID   ID                                                               Usage      |
|=========================================================================================|

Note: This works only with Secure Boot disabled. Users with Secure Boot enabled must wait for AlmaLinux to publish signed kmods for each kernel update.

Best regards

Henri

I did not mention in my previous comment that I have Secure Boot enabled on both of the systems.

According to NVidia’s own documentation

Assuming system with no previous NVidia packages nor conflicting repos:

# Necessary repos
sudo dnf install epel-release
sudo dnf config-manager --add-repo \
    https://developer.download.nvidia.com/compute/cuda/repos/rhel9/x86_64/cuda-rhel9.repo

# Choose stream (open, latest, not prebuilt)
sudo dnf module enable nvidia-driver:open-dkms

# Install both desktop and CUDA drivers
sudo dnf install nvidia-open

# Certificate from DKMS to firmware
sudo mokutil --import /var/lib/dkms/mok.pub

# A reboot and pre-boot action aro required to complete the import

Thanks for your reply and explanations ; dor the moment problem is solved !

It would benefit everyone if the AlmaLinux Team harmonized and made transparent the solutions regarding the integration of Nvidia/CUDA drivers and current and future kernels. Wait for secure migration from 9.7 to 10 in production server !

Regards

Henri

The lesson from this long day:
The problem wasn’t the NVIDIA driver—it was the source of the kmod. The AlmaLinux-NVIDIA repo publishes pre-compiled kmods that don’t keep up with kernel updates. The long-term solution is kmod-nvidia-latest-dkms from the official NVIDIA CUDA repo—it compiles locally and adapts to each kernel automatically.
Never again stuck on kernel versioning. :trophy:

Note:

  • The kmod-nvidia-latest-dkms is the “proprietary” version
    • Does not require GSP (GPU System Processor)
    • Does not support Blackwell or newer GPU’s
  • The kmod-nvidia-open-dkms is the “open” version
    • Requires GSP (introduced with Turing)
    • Cannot support older than Turing GPU’s

NVidia recommends the open version (where possible).

1 Like