Transient errors during package upgrade on AlmaLinux 9.7

My team has some tooling to create various hardened Linux images and we’ve been seeing transient errors on AlmaLinux 9.7 instances on Google Cloud Platform (GCP), especially while upgrading all packages (sudo dnf upgrade -y):

stderr

[Errno 2] No such file or directory: ‘/var/cache/dnf/baseos-09b2e93483836b7c/packages/kernel-5.14.0-611.30.1.el9_7.x86_64.rpm’

Usually this works fine but occasionally we’ll get an error like this. It’s quite a baffling situation and we can’t understand how to avoid these errors, even though they are transient. They prevent us from getting successful builds and we have to re-run jobs and cross our fingers, hoping it doesn’t happen again.

I’ve opened a support case with GCP but I’m wondering if the problem is in AlmaLinux.

Does anyone have any advice?

I’m thinking this is happening because of DNF Automatic and observed the following in /etc/dnf/automatic.conf:

download_updates = yes
apply_updates = yes

I am looking at potentially disabling automatic updates, maybe with systemctl stop dnf-automatic.timer. Thoughts? Does anyone sugest a better way to temporarily disable automatic updates?

This is likely due to a conflict between dnf-automatic and dnf upgrade running at the same time.

First, stop automatic updates during the build.

sudo systemctl stop dnf-automatic.timer