Installing NVIDIA driver Cuda drivers almalinux 9.5

Henri · March 5, 2025, 8:04am

I’m stuck on integrating NVIDIA drivers CUDA on almalinux 9.5 for a small local LLM project on DELL T340 server, RTX3080 gpu .
I’ve tried many installation modes and settings (Wiki almalinux, web comments about, Direct Nvidia drivers, …)
secure boot is disabled!
any idea please ?
$uname -a
Linux sirius 5.14.0-503.26.1.el9_5.x86_64 #1 SMP PREEMPT_DYNAMIC Mon Mar 3 05:56:39 EST 2025 x86_64 x86_64 x86_64 GNU/Linux
$dmesg | grep nvidia
$
$lsmod | grep nvidia
$
$sudo nvidia-smi
NVIDIA-SMI has failed because it couldn’t communicate with the NVIDIA driver. Make sure that the latest NVIDIA driver is installed and running.

$sudo dnf search nvidia-driver
Dernière vérification de l’expiration des métadonnées effectuée il y a 11:36:53 le mar. 04 mars 2025 21:00:15.
================================================================= Nom correspond exactement à : nvidia-driver ==================================================================
nvidia-driver.x86_64 : NVIDIA’s proprietary display driver for NVIDIA graphic cards
================================================================== Nom & Résumé correspond à : nvidia-driver ===================================================================
nvidia-driver-cuda.x86_64 : CUDA integration for nvidia-driver
nvidia-driver-cuda-libs.x86_64 : Libraries for nvidia-driver-cuda
nvidia-driver-cuda-libs.i686 : Libraries for nvidia-driver-cuda
nvidia-driver-libs.x86_64 : Libraries for nvidia-driver
nvidia-driver-libs.i686 : Libraries for nvidia-driver
======================================================================= Nom correspond à : nvidia-driver =======================================================================
libva-nvidia-driver.x86_64 : A VA-API implemention using NVIDIA’s NVDEC
nvidia-driver-assistant.noarch : Detect and install the best NVIDIA driver packages for the system
$
$
ls -l /lib/modules/6.13.5-1.el9.elrepo.x86_64/kernel/drivers
total 128
drwxr-xr-x. 5 root root 4096 27 févr. 20:57 acpi
drwxr-xr-x. 2 root root 4096 27 févr. 20:57 ata
drwxr-xr-x. 4 root root 32 27 févr. 20:56 base
drwxr-xr-x. 2 root root 24 27 févr. 20:57 bcma
drwxr-xr-x. 6 root root 4096 27 févr. 20:57 block
drwxr-xr-x. 2 root root 4096 27 févr. 20:57 bluetooth
drwxr-xr-x. 3 root root 17 27 févr. 20:56 bus
drwxr-xr-x. 2 root root 25 27 févr. 20:57 cdrom
drwxr-xr-x. 6 root root 192 27 févr. 20:57 char
drwxr-xr-x. 3 root root 4096 27 févr. 20:57 clk
drwxr-xr-x. 2 root root 50 27 févr. 20:57 counter
drwxr-xr-x. 2 root root 192 27 févr. 20:57 cpufreq
drwxr-xr-x. 10 root root 4096 27 févr. 20:57 crypto
drwxr-xr-x. 2 root root 49 27 févr. 20:57 cxl
drwxr-xr-x. 3 root root 103 27 févr. 20:57 dax
drwxr-xr-x. 2 root root 23 27 févr. 20:57 dca
drwxr-xr-x. 2 root root 43 27 févr. 20:57 devfreq
drwxr-xr-x. 11 root root 4096 27 févr. 20:57 dma
drwxr-xr-x. 2 root root 4096 27 févr. 20:57 edac
drwxr-xr-x. 2 root root 31 27 févr. 20:57 extcon
drwxr-xr-x. 3 root root 61 27 févr. 20:57 firmware
drwxr-xr-x. 2 root root 4096 27 févr. 20:57 gpio
drwxr-xr-x. 3 root root 17 27 févr. 20:56 gpu
drwxr-xr-x. 5 root root 8192 27 févr. 20:57 hid
drwxr-xr-x. 2 root root 74 27 févr. 20:57 hv
drwxr-xr-x. 4 root root 8192 27 févr. 20:57 hwmon
drwxr-xr-x. 4 root root 33 27 févr. 20:56 hwtracing
drwxr-xr-x. 5 root root 163 27 févr. 20:57 i2c
drwxr-xr-x. 14 root root 4096 27 févr. 20:57 iio
drwxr-xr-x. 6 root root 49 27 févr. 20:56 infiniband
drwxr-xr-x. 12 root root 4096 27 févr. 20:57 input
drwxr-xr-x. 3 root root 47 27 févr. 20:57 iommu
drwxr-xr-x. 4 root root 35 27 févr. 20:56 isdn
drwxr-xr-x. 6 root root 4096 27 févr. 20:57 leds
drwxr-xr-x. 2 root root 60 27 févr. 20:57 mailbox
drwxr-xr-x. 4 root root 4096 27 févr. 20:57 md
drwxr-xr-x. 13 root root 152 27 févr. 20:56 media
$ ls -l /lib/modules/6.13.5-1.el9.elrepo.x86_64/kernel/drivers/gpu/drm
total 508
drwxr-xr-x. 4 root root 34 27 févr. 20:56 amd
drwxr-xr-x. 3 root root 21 27 févr. 20:56 arm
drwxr-xr-x. 2 root root 23 27 févr. 20:57 ast
drwxr-xr-x. 2 root root 38 27 févr. 20:57 display
-rwxr–r–. 1 root root 9496 27 févr. 19:37 drm_buddy.ko.xz
-rwxr–r–. 1 root root 11676 27 févr. 19:37 drm_client_lib.ko.xz
-rwxr–r–. 1 root root 19108 27 févr. 19:37 drm_dma_helper.ko.xz
-rwxr–r–. 1 root root 4808 27 févr. 19:37 drm_exec.ko.xz
-rwxr–r–. 1 root root 18208 27 févr. 19:37 drm_gpuvm.ko.xz
-rwxr–r–. 1 root root 88060 27 févr. 19:37 drm_kms_helper.ko.xz
-rwxr–r–. 1 root root 260724 27 févr. 19:37 drm.ko.xz
-rwxr–r–. 1 root root 23048 27 févr. 19:37 drm_mipi_dbi.ko.xz
-rwxr–r–. 1 root root 1800 27 févr. 19:37 drm_panel_backlight_quirks.ko.xz
-rwxr–r–. 1 root root 20292 27 févr. 19:37 drm_shmem_helper.ko.xz
-rwxr–r–. 1 root root 5520 27 févr. 19:37 drm_suballoc_helper.ko.xz
-rwxr–r–. 1 root root 15292 27 févr. 19:37 drm_ttm_helper.ko.xz
-rwxr–r–. 1 root root 14016 27 févr. 19:37 drm_vram_helper.ko.xz
drwxr-xr-x. 2 root root 23 27 févr. 20:57 gud
drwxr-xr-x. 2 root root 30 27 févr. 20:57 hyperv
drwxr-xr-x. 2 root root 88 27 févr. 20:57 i2c
drwxr-xr-x. 2 root root 43 27 févr. 20:57 i915
drwxr-xr-x. 2 root root 27 27 févr. 20:57 mgag200
drwxr-xr-x. 2 root root 27 27 févr. 20:57 nouveau
drwxr-xr-x. 2 root root 23 27 févr. 20:57 qxl
drwxr-xr-x. 2 root root 26 27 févr. 20:57 radeon
drwxr-xr-x. 2 root root 29 27 févr. 20:57 scheduler
drwxr-xr-x. 2 root root 4096 27 févr. 20:57 tiny
drwxr-xr-x. 2 root root 23 27 févr. 20:57 ttm
drwxr-xr-x. 2 root root 23 27 févr. 20:57 udl
drwxr-xr-x. 2 root root 29 27 févr. 20:57 vboxvideo
drwxr-xr-x. 2 root root 24 27 févr. 20:57 vgem
drwxr-xr-x. 2 root root 30 27 févr. 20:57 virtio
drwxr-xr-x. 2 root root 24 27 févr. 20:57 vkms
drwxr-xr-x. 2 root root 26 27 févr. 20:57 vmwgfx
drwxr-xr-x. 2 root root 33 27 févr. 20:57 xen
$
drwxr-xr-x. 4 root root 30 27 févr. 20:56 memstick
drwxr-xr-x. 3 root root 20 27 févr. 20:56 message
drwxr-xr-x. 2 root root 4096 27 févr. 20:57 mfd
drwxr-xr-x. 12 root root 4096 27 févr. 20:57 misc
drwxr-xr-x. 4 root root 30 27 févr. 20:56 mmc
drwxr-xr-x. 6 root root 124 27 févr. 20:57 mtd
drwxr-xr-x. 2 root root 28 27 févr. 20:57 mux
drwxr-xr-x. 25 root root 4096 27 févr. 20:57 net
drwxr-xr-x. 4 root root 72 27 févr. 20:57 ntb
drwxr-xr-x. 2 root root 160 27 févr. 20:57 nvdimm
drwxr-xr-x. 4 root root 32 27 févr. 20:56 nvme
drwxr-xr-x. 2 root root 83 27 févr. 20:57 of
drwxr-xr-x. 2 root root 79 27 févr. 20:57 parport
drwxr-xr-x. 6 root root 90 27 févr. 20:57 pci
drwxr-xr-x. 11 root root 4096 27 févr. 20:57 phy
drwxr-xr-x. 4 root root 131 27 févr. 20:57 pinctrl
drwxr-xr-x. 5 root root 47 27 févr. 20:56 platform
drwxr-xr-x. 3 root root 20 27 févr. 20:56 power
drwxr-xr-x. 2 root root 65 27 févr. 20:57 powercap
drwxr-xr-x. 3 root root 21 27 févr. 20:56 pps
drwxr-xr-x. 2 root root 94 27 févr. 20:57 ptp
drwxr-xr-x. 2 root root 4096 27 févr. 20:57 pwm
drwxr-xr-x. 3 root root 17 27 févr. 20:56 ras
drwxr-xr-x. 2 root root 4096 27 févr. 20:57 regulator
drwxr-xr-x. 2 root root 4096 27 févr. 20:57 rtc
drwxr-xr-x. 32 root root 4096 27 févr. 20:57 scsi
drwxr-xr-x. 3 root root 18 27 févr. 20:56 soc
drwxr-xr-x. 2 root root 162 27 févr. 20:57 soundwire
drwxr-xr-x. 2 root root 4096 27 févr. 20:57 spi
drwxr-xr-x. 2 root root 23 27 févr. 20:57 ssb
drwxr-xr-x. 3 root root 23 27 févr. 20:56 staging
drwxr-xr-x. 5 root root 4096 27 févr. 20:57 target
drwxr-xr-x. 2 root root 23 27 févr. 20:57 tee
drwxr-xr-x. 3 root root 78 27 févr. 20:57 thermal
drwxr-xr-x. 3 root root 79 27 févr. 20:57 tty
drwxr-xr-x. 2 root root 4096 27 févr. 20:57 uio
drwxr-xr-x. 14 root root 161 27 févr. 20:56 usb
drwxr-xr-x. 9 root root 131 27 févr. 20:57 vdpa
drwxr-xr-x. 4 root root 77 27 févr. 20:57 vfio
drwxr-xr-x. 2 root root 166 27 févr. 20:57 vhost
drwxr-xr-x. 4 root root 36 3 mars 18:32 video
drwxr-xr-x. 4 root root 40 27 févr. 20:56 virt
drwxr-xr-x. 2 root root 162 27 févr. 20:57 virtio
drwxr-xr-x. 2 root root 4096 27 févr. 20:57 watchdog
drwxr-xr-x. 3 root root 4096 27 févr. 20:57 xen
many thanks
HI

jlehtone · March 5, 2025, 11:55am

You have an application that needs CUDA runtime libraries to run (and possibly devel to build).
The CUDA runtime requires NVidia’s GPU driver (the Nouveau that is included with Alma will not do).

In principle any version of NVidia’s packages should do (NVidia’s, ELRepo’s, RPM Fusion’s) but it is easiest to use NVidia’s as they are in same repo as the CUDA packages.

One needs EPEL, so dnf install epel-release
EPEL may need CRB, so crb --enable
Then the CUDA repo. I have:

# cat /etc/yum.repos.d/cuda-rhel9.repo 
[cuda-rhel9-x86_64]
baseurl = https://developer.download.nvidia.com/compute/cuda/repos/rhel9/x86_64
enabled = 1
gpgcheck = 1
gpgkey = https://developer.download.nvidia.com/compute/cuda/repos/rhel9/x86_64/D42D0685.pub
name = cuda-rhel9-x86_64

With those in place, first make sure that “stream” is enabled:

dnf module enable nvidia-driver:nvidia-driver

Then install driver-related packages and the CUDA:

dnf install nvidia-driver nvidia-driver-cuda nvidia-kmod-common nvidia-modprobe nvidia-settings nvidia-xconfig dnf-plugin-nvidia kernel-devel-matched dkms cuda-toolkit-12-8

Note that this list of packages contains cuda-toolkit-12-8 that is currently latests CUDA on the repo. You can see other options with:

dnf list cuda-toolkit-??-?

Henri · March 6, 2025, 11:03am

HI,
Thank you for your response and your appreciated help.
The recommendations that you have sent me have been carried out many times and on different levels, notably the direct integration of the manual procedure of the latest NVIDIA driver (NVIDIA-Linux-x86_64-570.124.04.run).
Always at the same point the driver is present but inactive!
Some extracted system elements:
$ Linux sirius 6.13.5-1.el9.elrepo.x86_64 #1 SMP PREEMPT_DYNAMIC Thu Feb 27 12:59:45 EST 2025 x86_64 x86_64 x86_64 GNU/Linux
actualy display with matrox vgEmbedded Video motherboard
$ nvidia-smi
NVIDIA-SMI has failed because it couldn’t communicate with the NVIDIA driver. Make sure that the latest NVIDIA driver is installed and running.
ars 06 11:32:44 sirius systemd-modules-load[332]: Failed to find module ‘nvidia-fs’
mars 06 11:32:48 sirius systemd-modules-load[760]: Failed to find module ‘nvidia-fs’
mars 06 11:32:50 sirius systemd[1]: Started nvidia-powerd service.
mars 06 11:32:50 sirius /usr/bin/nvidia-powerd[1253]: nvidia-powerd version:1.0(build 1)
mars 06 11:32:50 sirius /usr/bin/nvidia-powerd[1253]: Allocate client failed 89
mars 06 11:32:50 sirius /usr/bin/nvidia-powerd[1253]: Failed to initialize RM Client
mars 06 11:32:50 sirius systemd[1]: nvidia-powerd.service: Deactivated successfully.
mars 06 11:32:51 sirius akmods[1219]: Checking kmods exist for 6.13.5-1.el9.elrepo.x86_64Warning: Could not determine what package owns /lib/modules/6.13.5-1.el9.elrepo.x86_64/extra/nvidia/ [ OK ]
mars 06 11:32:51 sirius akmods[1219]: Checking kmods exist for 5.14.0-503.26.1.el9_5.x86_64Warning: Could not determine what package owns /lib/modules/5.14.0-503.26.1.el9_5.x86_64/extra/nvidia/ [ OK ]
mars 06 11:32:51 sirius dkms[1939]: Deprecated feature: REMAKE_INITRD (/var/lib/dkms/nvidia-fs/2.24.2/source/dkms.conf)
mars 06 11:32:51 sirius dkms[2005]: Deprecated feature: REMAKE_INITRD (/var/lib/dkms/nvidia-fs/2.24.2/source/dkms.conf)
mars 06 11:32:51 sirius dkms[1229]: Autoinstall of module nvidia-fs/2.24.2 for kernel 6.13.5-1.el9.elrepo.x86_64 (x86_64)
$ dkms status
/usr/bin/which: no ofed_info in (/home/henri/.local/bin:/home/henri/bin:/usr/share/Modules/bin:/usr/local/bin:/usr/local/sbin:/usr/bin:/usr/sbin)
Deprecated feature: REMAKE_INITRD (/var/lib/dkms/nvidia-fs/2.24.2/source/dkms.conf)
nvidia/570.124.06, 5.14.0-503.23.2.el9_5.x86_64, x86_64: installed
nvidia/570.124.06, 5.14.0-503.26.1.el9_5.x86_64, x86_64: installed
nvidia/570.124.06, 6.13.5-1.el9.elrepo.x86_64, x86_64: installed (Differences between built and installed modules)
nvidia-fs/2.24.2, 5.14.0-503.26.1.el9_5.x86_64, x86_64: installed
any idea ?
Best regards
Henri

jlehtone · March 6, 2025, 11:22am

I have not touched that in years. It is a “source install”. The package manager (dnf) does not know anything about source install and thus one can trample the files of the other. It is more cansistent, “managed”, to have all installs from RPM packages.

Now you reveal that you have also non-AlmaLinux kernel from ELRepo. That does not make things simpler. I don’t use those kernels anywhere, so cannot advice about them.

Something to gather information with:

cat /proc/cmdline 
cat /etc/kernel/cmdline 
grep CMDLINE /etc/default/grub
grep -r options /boot/loader
symlinks -r /lib/modules
find /lib/modules -type f -name "nvidia*"

Henri · March 6, 2025, 1:40pm

Jlehtone,
I want to use the GPU card not for display but to set up an LLM project located on the server. The card still needs to be recognized to implement cuda and other GPU tools !
I thought that making a standard NVIDIA choice such as the RTX 3080 would be easier than AMD or other!
Thanks again for your valuable help.
Here is your additional technical request :
$cat /proc/cmdline
BOOT_IMAGE=(hd0,gpt2)/vmlinuz-6.13.5-1.el9.elrepo.x86_64 root=/dev/mapper/almalinux-root ro crashkernel=1G-4G:192M,4G-64G:256M,64G-:512M resume=/dev/mapper/almalinux-swap rd.lvm.lv=almalinux/root rd.lvm.lv=almalinux/swap rhgb quiet rd.driver.blacklist=nouveau
GRUB_DEFAULT=0 audit=1 selinux=1
$ cat /etc/kernel/cmdline
root=/dev/mapper/almalinux-root ro crashkernel=1G-4G:192M,4G-64G:256M,64G-:512M resume=/dev/mapper/almalinux-swap rd.lvm.lv=almalinux/root rd.lvm.lv=almalinux/swap rhgb quiet r
d.driver.blacklist=nouveau GRUB_DEFAULT=0 audit=1 selinux=1
$ grep CMDLINE /etc/default/grub
GRUB_CMDLINE_LINUX=“crashkernel=1G-4G:192M,4G-64G:256M,64G-:512M resume=/dev/mapper/almalinux-swap rd.lvm.lv=almalinux/root rd.lvm.lv=almalinux/swap rhgb quiet rd.driver.blackl
ist=nouveau GRUB_DEFAULT=0 audit=1 selinux=1”
$ grep -r options /boot/loader
/boot/loader/entries/c304f152df184a6bb52c47dcc552ba3e-0-rescue.conf:options root=/dev/mapper/almalinux-root ro crashkernel=1G-4G:192M,4G-64G:256M,64G-:512M resume=/dev/mapper/a
lmalinux-swap rd.lvm.lv=almalinux/root rd.lvm.lv=almalinux/swap rhgb quiet rd.driver.blacklist=nouveau GRUB_DEFAULT=0 audit=1 selinux=1
/boot/loader/entries/c304f152df184a6bb52c47dcc552ba3e-5.14.0-503.23.1.el9_5.x86_64.conf:options root=/dev/mapper/almalinux-root ro crashkernel=1G-4G:192M,4G-64G:256M,64G-:512M
resume=/dev/mapper/almalinux-swap rd.lvm.lv=almalinux/root rd.lvm.lv=almalinux/swap rhgb quiet $tuned_params rd.driver.blacklist=nouveau GRUB_DEFAULT=0 audit=1 selinux=1
/boot/loader/entries/c304f152df184a6bb52c47dcc552ba3e-5.14.0-503.23.2.el9_5.x86_64.conf:options root=/dev/mapper/almalinux-root ro crashkernel=1G-4G:192M,4G-64G:256M,64G-:512M
resume=/dev/mapper/almalinux-swap rd.lvm.lv=almalinux/root rd.lvm.lv=almalinux/swap rhgb quiet $tuned_params rd.driver.blacklist=nouveau GRUB_DEFAULT=0 audit=1 selinux=1
/boot/loader/entries/c304f152df184a6bb52c47dcc552ba3e-6.13.5-1.el9.elrepo.x86_64.conf:options root=/dev/mapper/almalinux-root ro crashkernel=1G-4G:192M,4G-64G:256M,64G-:512M re
sume=/dev/mapper/almalinux-swap rd.lvm.lv=almalinux/root rd.lvm.lv=almalinux/swap rhgb quiet $tuned_params rd.driver.blacklist=nouveau GRUB_DEFAULT=0 audit=1 selinux=1
/boot/loader/entries/c304f152df184a6bb52c47dcc552ba3e-5.14.0-503.26.1.el9_5.x86_64.conf:options root=/dev/mapper/almalinux-root ro crashkernel=1G-4G:192M,4G-64G:256M,64G-:512M
resume=/dev/mapper/almalinux-swap rd.lvm.lv=almalinux/root rd.lvm.lv=almalinux/swap rhgb quiet rd.driver.blacklist=nouveau $tuned_params GRUB_DEFAULT=0 audit=1 selinux=1
$ symlinks -r /lib/modules
absolute: /lib/modules/5.14.0-503.23.1.el9_5.x86_64/build → /usr/src/kernels/5.14.0-503.23.1.el9_5.x86_64
absolute: /lib/modules/5.14.0-503.23.1.el9_5.x86_64/weak-updates/kmod-kvdo/vdo/kvdo.ko → /lib/modules/5.14.0-503.11.1.el9_5.x86_64/extra/kmod-kvdo/vdo/kvdo.ko
absolute: /lib/modules/5.14.0-503.23.1.el9_5.x86_64/weak-updates/nvidia-drm.ko.xz → /lib/modules/5.14.0-503.26.1.el9_5.x86_64/extra/nvidia-drm.ko.xz
absolute: /lib/modules/5.14.0-503.23.1.el9_5.x86_64/weak-updates/nvidia-modeset.ko.xz → /lib/modules/5.14.0-503.26.1.el9_5.x86_64/extra/nvidia-modeset.ko.xz
absolute: /lib/modules/5.14.0-503.23.1.el9_5.x86_64/weak-updates/nvidia-peermem.ko.xz → /lib/modules/5.14.0-503.26.1.el9_5.x86_64/extra/nvidia-peermem.ko.xz
absolute: /lib/modules/5.14.0-503.23.1.el9_5.x86_64/weak-updates/nvidia-uvm.ko.xz → /lib/modules/5.14.0-503.26.1.el9_5.x86_64/extra/nvidia-uvm.ko.xz
absolute: /lib/modules/5.14.0-503.23.1.el9_5.x86_64/weak-updates/nvidia.ko.xz → /lib/modules/5.14.0-503.26.1.el9_5.x86_64/extra/nvidia.ko.xz
absolute: /lib/modules/5.14.0-503.23.2.el9_5.x86_64/build → /usr/src/kernels/5.14.0-503.23.2.el9_5.x86_64
absolute: /lib/modules/5.14.0-503.23.2.el9_5.x86_64/weak-updates/kmod-kvdo/vdo/kvdo.ko → /lib/modules/5.14.0-503.11.1.el9_5.x86_64/extra/kmod-kvdo/vdo/kvdo.ko
absolute: /lib/modules/5.14.0-503.23.2.el9_5.x86_64/weak-updates/nvidia-drm.ko.xz → /lib/modules/5.14.0-503.26.1.el9_5.x86_64/extra/nvidia-drm.ko.xz
absolute: /lib/modules/5.14.0-503.23.2.el9_5.x86_64/weak-updates/nvidia-modeset.ko.xz → /lib/modules/5.14.0-503.26.1.el9_5.x86_64/extra/nvidia-modeset.ko.xz
absolute: /lib/modules/5.14.0-503.23.2.el9_5.x86_64/weak-updates/nvidia-peermem.ko.xz → /lib/modules/5.14.0-503.26.1.el9_5.x86_64/extra/nvidia-peermem.ko.xz
absolute: /lib/modules/5.14.0-503.23.2.el9_5.x86_64/weak-updates/nvidia-uvm.ko.xz → /lib/modules/5.14.0-503.26.1.el9_5.x86_64/extra/nvidia-uvm.ko.xz
absolute: /lib/modules/5.14.0-503.23.2.el9_5.x86_64/weak-updates/nvidia.ko.xz → /lib/modules/5.14.0-503.26.1.el9_5.x86_64/extra/nvidia.ko.xz
absolute: /lib/modules/6.13.5-1.el9.elrepo.x86_64/build → /usr/src/kernels/6.13.5-1.el9.elrepo.x86_64
absolute: /lib/modules/5.14.0-503.26.1.el9_5.x86_64/build → /usr/src/kernels/5.14.0-503.26.1.el9_5.x86_64
absolute: /lib/modules/5.14.0-503.26.1.el9_5.x86_64/weak-updates/kmod-kvdo/vdo/kvdo.ko → /lib/modules/5.14.0-503.11.1.el9_5.x86_64/extra/kmod-kvdo/vdo/kvdo.ko
absolute: /lib/modules/5.14.0-503.26.1.el9_5.x86_64/weak-updates/nvidia-drm.ko.xz → /lib/modules/5.14.0-503.23.2.el9_5.x86_64/extra/nvidia-drm.ko.xz
absolute: /lib/modules/5.14.0-503.26.1.el9_5.x86_64/weak-updates/nvidia-modeset.ko.xz → /lib/modules/5.14.0-503.23.2.el9_5.x86_64/extra/nvidia-modeset.ko.xz
absolute: /lib/modules/5.14.0-503.26.1.el9_5.x86_64/weak-updates/nvidia-peermem.ko.xz → /lib/modules/5.14.0-503.23.2.el9_5.x86_64/extra/nvidia-peermem.ko.xz
absolute: /lib/modules/5.14.0-503.26.1.el9_5.x86_64/weak-updates/nvidia-uvm.ko.xz → /lib/modules/5.14.0-503.23.2.el9_5.x86_64/extra/nvidia-uvm.ko.xz
absolute: /lib/modules/5.14.0-503.26.1.el9_5.x86_64/weak-updates/nvidia.ko.xz → /lib/modules/5.14.0-503.23.2.el9_5.x86_64/extra/nvidia.ko.xz
$ find /lib/modules -type f -name “nvidia*”
/lib/modules/5.14.0-503.23.2.el9_5.x86_64/extra/nvidia.ko.xz
/lib/modules/5.14.0-503.23.2.el9_5.x86_64/extra/nvidia-modeset.ko.xz
/lib/modules/5.14.0-503.23.2.el9_5.x86_64/extra/nvidia-drm.ko.xz
/lib/modules/5.14.0-503.23.2.el9_5.x86_64/extra/nvidia-uvm.ko.xz
/lib/modules/5.14.0-503.23.2.el9_5.x86_64/extra/nvidia-peermem.ko.xz
/lib/modules/6.13.5-1.el9.elrepo.x86_64/extra/nvidia.ko
/lib/modules/6.13.5-1.el9.elrepo.x86_64/extra/nvidia-uvm.ko
/lib/modules/6.13.5-1.el9.elrepo.x86_64/extra/nvidia-modeset.ko
/lib/modules/6.13.5-1.el9.elrepo.x86_64/extra/nvidia-drm.ko
/lib/modules/6.13.5-1.el9.elrepo.x86_64/extra/nvidia-peermem.ko
/lib/modules/6.13.5-1.el9.elrepo.x86_64/extra/nvidia.ko.xz
/lib/modules/6.13.5-1.el9.elrepo.x86_64/extra/nvidia-modeset.ko.xz
/lib/modules/6.13.5-1.el9.elrepo.x86_64/extra/nvidia-drm.ko.xz
/lib/modules/6.13.5-1.el9.elrepo.x86_64/extra/nvidia-uvm.ko.xz
/lib/modules/6.13.5-1.el9.elrepo.x86_64/extra/nvidia-peermem.ko.xz
/lib/modules/5.14.0-503.26.1.el9_5.x86_64/kernel/drivers/platform/x86/nvidia-wmi-ec-backlight.ko.xz
/lib/modules/5.14.0-503.26.1.el9_5.x86_64/extra/nvidia.ko.xz
/lib/modules/5.14.0-503.26.1.el9_5.x86_64/extra/nvidia-modeset.ko.xz
/lib/modules/5.14.0-503.26.1.el9_5.x86_64/extra/nvidia-drm.ko.xz
/lib/modules/5.14.0-503.26.1.el9_5.x86_64/extra/nvidia-uvm.ko.xz
/lib/modules/5.14.0-503.26.1.el9_5.x86_64/extra/nvidia-peermem.ko.xz
/lib/modules/5.14.0-503.26.1.el9_5.x86_64/extra/nvidia-fs.ko.xz

Best regard
Henri

jlehtone · March 6, 2025, 3:02pm

Yes. The LLM needs CUDA libraries and the CUDA libs need the GPU drivers. That is a fact.

If you don’t want to use that GPU for displaying anything, then you have to configure the Window Manager / Display Manager / something to use the other GPU even when the NVidia GPU is available. I have never done such config.

PS. You can post your outputs with code tags (the </> button. Makes reading easier.

Henri · March 6, 2025, 3:39pm

This is already the case my display works with the Matrox VGA motherboard of the DELL server. But unfortunately the NVIDIA card without the functional drivers cannot work even in pure Cuda calculation! The card does not respond; no active cooling!
I will continue my investigations to try to understand this problem a little better!
Thanks again for your help
PS: if you know a Unix driver/kernel specialist let me know
Best Regards
Henri

jlehtone · March 6, 2025, 6:09pm

You seem to have kernel module files. Therefore,

modinfo nvidia

should tell about that module, and

modprobe nvidia

may or may not say something, just like the

lsmod | grep nvidia

after that.

Henri · March 6, 2025, 7:37pm

I I think I have narrowed down the scope of the problem, the problem seems to be at the level of the integration of the module into the kernel, the “dkms” command is in error: (consistent with the result of the commands modprobe modinfo lsmode /nvidia )
journalctl -xeu dkms.service
Support: Help and Support | AlmaLinux Wiki
An ExecStart= process belonging to unit dkms.service has exited.
The process’ exit code is ‘exited’ and its exit status is 11.
mars 06 18:50:09 sirius systemd[1]: dkms.service: Failed with result ‘exit-code’.
Subject: Unit failed
Defined-By: systemd
Support: Help and Support | AlmaLinux Wiki
The unit dkms.service has entered the ‘failed’ state with result ‘exit-code’.
mars 06 18:50:09 sirius systemd[1]: Failed to start Builds and install new kernel modules through DKMS.
Subject: L’unité (unit) dkms.service a échoué
Defined-By: systemd
Support: Help and Support | AlmaLinux Wiki
When i tried to restart dkms here result
$ sudo systemctl status dkms
× dkms.service - Builds and install new kernel modules through DKMS
Loaded: loaded (/usr/lib/systemd/system/dkms.service; enabled; preset: disabled)
Active: failed (Result: exit-code) since Thu 2025-03-06 18:50:09 CET; 55s ago
Docs: man:dkms(8)
Process: 42747 ExecStart=/usr/sbin/dkms autoinstall --verbose --kernelver 6.13.5-1.el9.elrepo.x86_64 (code=exited, status=11)
Main PID: 42747 (code=exited, status=11)
CPU: 37.108s

mars 06 18:50:09 sirius dkms[50245]: Error! Bad return status for module build on kernel: 6.13.5-1.el9.elrepo.x86_64 (x86_64)
mars 06 18:50:09 sirius dkms[50245]: Consult /var/lib/dkms/nvidia-fs/2.24.2/build/make.log for more information.
mars 06 18:50:09 sirius dkms[42747]: Autoinstall on 6.13.5-1.el9.elrepo.x86_64 succeeded for module(s) nvidia.
mars 06 18:50:09 sirius dkms[42747]: Autoinstall on 6.13.5-1.el9.elrepo.x86_64 failed for module(s) nvidia-fs(10).
mars 06 18:50:09 sirius dkms[50246]: Error! One or more modules failed to install during autoinstall.
mars 06 18:50:09 sirius dkms[50246]: Refer to previous errors for more information.
mars 06 18:50:09 sirius systemd[1]: dkms.service: Main process exited, code=exited, status=11/n/a
mars 06 18:50:09 sirius systemd[1]: dkms.service: Failed with result ‘exit-code’.
mars 06 18:50:09 sirius systemd[1]: Failed to start Builds and install new kernel modules through DKMS.
mars 06 18:50:09 sirius systemd[1]: dkms.service: Consumed 37.108s CPU time.
verification on make error file designed “/var/lib/dkms/nvidia-fs/2.24.2/build/make.log”
Picking NVIDIA driver sources from NVIDIA_SRC_DIR=/usr/src/nvidia-570.124.06/nvidia. If that does not meet your expectation, you might have a stale driver still around and that might cause problems.
‘/lib/modules/6.13.5-1.el9.elrepo.x86_64/extra/nvidia.ko.xz’ → ‘./nvidia.ko.xz’
now question is why make code fail ?
Regards
Henri

jlehtone · March 6, 2025, 8:17pm

Henri:

mars 06 18:50:09 sirius dkms[50245]: Error! Bad return status for module build on kernel: 6.13.5-1.el9.elrepo.x86_64 (x86_64)
mars 06 18:50:09 sirius dkms[50245]: Consult /var/lib/dkms/nvidia-fs/2.24.2/build/make.log for more information.

That error is for the kernel-6.13.5-1.el9.elrepo.x86_64. As said, it is very different from the AlmaLinux kernels. I would remove that kernel completely to see whether the normal kernels “behave”.

Henri · March 10, 2025, 9:56am

HI,
After laborious and fruitless tests and extended research, I confirm that the Dell T340 is inoperable with GPU cards or other GPU(Tested on my RTX 3060/80 gpu!). The matrox devalidated in the bios (BIOS_5HWMC_LN64_2.18.0) does not solve the problem; virtual devalidation, the display is still present with the GPU on the pcie 2 slot.

Curious that the integrated matrox graphics card cannot actually be disabled by the bios ! [ Dell’s likely will on the T340?]

the linux lspci command attests to the non-recognition of the card! Big disappointment. I am continuing the project and you are directing me towards an assembled hardware solution probably allowing an optimization of the hardware cost!
Thanks for your Help
best regards

Henri

jlehtone · March 10, 2025, 10:39am

According to https://i.dell.com/sites/csdocuments/Product_Docs/en/dell-emc-poweredge-t340-tech-guide.pdf the iDRAC is wired to the Matrox GPU. Some of the Xeon E-2200 CPU’s had IGP, but even the use of IGP is not possible on T340. In other words, if one wants to see the OS output via iDRAC, then the output has to go to the Matrox.

It should be possible to have both Matrox and NVidia drivers (loaded), and force output to use the Matrox. I just don’t know what kind of config is required for that.

More importantly, the T340 is listed to have:

“Single or Dual Redundant 495W power supply or single 350W cabled power supply”

The recommended PSU for RTX 3080 is 750 W, and the card alone consumes some 350 W. It does not look like that the T340 would have been built to host powerful discrete GPU’s.

Henri · March 10, 2025, 5:15pm

I have tested a lot of BIOS modifications without success. I have two 450W power supplies active and online. This Bios/UEFI does not take into account the enumeration of the RTX3080 references on the PCIE X16 slot.2 This card works and has been tested on a rather basic ASUS card!
On first pass enumérating
Pcie Slot 1 Empty ( yes correct empty )
Pcie Slot 2 Card detected ( Rtx 3080 )
Pcie Slot 3 Card detected ( Sound blaster )
Pcie Slot 4 Card detected ( Dell Raid )
in the continuation of the test Pcie bus test indicate: Bus 02: link not reported PCIE SLOT 02 empty"

I have tried a lot of BIOS modifications without success. I have two 450W power supplies active and online. This Bios/UEFI does not take into account the enumeration of the references of the RTX3080 on the PCIE 4 slot. This card works and has been tested on a rather basic ASUS card!
in the continuation of the test
I will try to contact DELL and ask them about the problem, and see if they can offer me a solution!
Regards
Henri

Henri · March 14, 2025, 5:36pm

for the community DEll users this can be perharps useful !

Yay, Good news !!!
I added a dedicated power supply to GPU, connecting the card’s 2x8 Y cable.
The BIOS shell sees the PCIe slot occupied as well as the named card.
I can now say that, with a few hardware modifications, the Dell T340 supports GPU cards, in this case the NVIDIA RTX 3080.
Now the LLM/CUDA project on ALMALINUX I had envisioned can begin…
Many thanks for the help with the specifics of dual power supplies for GPU cards.
Regards;
Sincerely,
Henri

Now :
basic commands on UEFI Shell pre boot indicates :
drivers : 21D 00060009 ? N N 0 0 NVIDIA GPU UEFI Driver PciRoot(0x0)/Pci(0x1,0x0)/Pci(0x0,0x0)/nvgop-ga1xx
pci: 00 06 00 00 ==> Multimedia Device - Mixed mode device
Vendor 1102 Device 0012 Prog Interface 0
Type=9, Handle=0x901
Dump Structure as:
Index=23,Length=0x1E,Addr=0x6A806322
00000000: 09 11 01 09 01 B6 0B 04-04 02 00 04 01 00 00 01 …
00000010: 00 50 43 49 65 20 53 6C-6F 74 20 32 00 00 .PCIe Slot 2…
Structure Type: System Slots
Format part Len : 17
Structure Handle: 2305
SlotDesignation: PCIe Slot 2
System Slot Type: PCI Express Gen 3 X16
System Slot Data Bus Width: 8x or x8
System Slot Current Usage: In use
System Slot Length: Long Length
System Slot Type: PCI Express Gen 3 X16
Slot Id: the value present in the Slot Number field of the PCI Interrupt Routing table entry that is associated with this slot is: 2
Slot characteristics 1: Provides 3.3 Volts
Slot characteristics 2: PCI slot supports Power Management Enable (PME#) signal
SegmentGroupNum: 0x0
BusNum: 0x1
DevFuncNum: 0x0
Almalinux : $ lspci
00:00.0 Host bridge: Intel Corporation 8th Gen Core Processor Host Bridge/DRAM Registers (rev 07)
00:01.0 PCI bridge: Intel Corporation 6th-10th Gen Core Processor PCIe Controller (x16) (rev 07)
00:01.1 PCI bridge: Intel Corporation Xeon E3-1200 v5/E3-1500 v5/6th Gen Core Processor PCIe Controller (x8) (rev 07)
00:08.0 System peripheral: Intel Corporation Xeon E3-1200 v5/v6 / E3-1500 v5 / 6th/7th/8th Gen Core Processor Gaussian Mixture Model
00:12.0 Signal processing controller: Intel Corporation Cannon Lake PCH Thermal Controller (rev 10)
00:14.0 USB controller: Intel Corporation Cannon Lake PCH USB 3.1 xHCI Host Controller (rev 10)
00:14.2 RAM memory: Intel Corporation Cannon Lake PCH Shared SRAM (rev 10)
00:16.0 Communication controller: Intel Corporation Cannon Lake PCH HECI Controller (rev 10)
00:16.4 Communication controller: Intel Corporation Cannon Lake PCH HECI Controller #2 (rev 10)
00:17.0 SATA controller: Intel Corporation Cannon Lake PCH SATA AHCI Controller (rev 10)
00:1c.0 PCI bridge: Intel Corporation Cannon Lake PCH PCI Express Root Port #1 (rev f0)
00:1c.1 PCI bridge: Intel Corporation Cannon Lake PCH PCI Express Root Port #2 (rev f0)
00:1c.7 PCI bridge: Intel Corporation Cannon Lake PCH PCI Express Root Port #8 (rev f0)
00:1d.0 PCI bridge: Intel Corporation Cannon Lake PCH PCI Express Root Port #9 (rev f0)
00:1e.0 Communication controller: Intel Corporation Cannon Lake PCH Serial IO UART Host Controller (rev 10)
00:1f.0 ISA bridge: Intel Corporation Cannon Point-LP LPC Controller (rev 10)
00:1f.4 SMBus: Intel Corporation Cannon Lake PCH SMBus Controller (rev 10)
00:1f.5 Serial bus controller: Intel Corporation Cannon Lake PCH SPI Controller (rev 10)
01:00.0 VGA compatible controller: NVIDIA Corporation GA102 [GeForce RTX 3080] (rev a1)
01:00.1 Audio device: NVIDIA Corporation GA102 High Definition Audio Controller (rev a1)
03:00.0 PCI bridge: PLDA PCI Express Bridge (rev 02)
04:00.0 VGA compatible controller: Matrox Electronics Systems Ltd. Integrated Matrox G200eW3 Graphics Controller (rev 04)
05:00.0 Ethernet controller: Broadcom Inc. and subsidiaries NetXtreme BCM5720 Gigabit Ethernet PCIe
05:00.1 Ethernet controller: Broadcom Inc. and subsidiaries NetXtreme BCM5720 Gigabit Ethernet PCIe
06:00.0 Audio device: Creative Labs CA0132 Sound Core3D [Sound Blaster Recon3D / Z-Series / Sound BlasterX AE-5 Plus] (rev 01)
07:00.0 RAID bus controller: Broadcom / LSI MegaRAID SAS-3 3108 [Invader] (rev 02)