Unable to detect release version

gran061 · May 27, 2024, 8:21am

Good Day

I have three Almalinux 8 servers, and they all three are running on 8.9. Out of the three, two seem to fail when I run yum/dnf update with the following error:

yum update
Unable to detect release version (use '--releasever' to specify release version)
AlmaLinux $releasever - BaseOS                                                                                                                                 49  B/s | 273  B     00:05    
Errors during downloading metadata for repository 'baseos':
  - Status code: 404 for https://mirrors.almalinux.org/mirrorlist/$releasever/baseos (IP: 52.200.132.63)
  - Status code: 404 for https://mirrors.almalinux.org/mirrorlist/$releasever/baseos (IP: 44.193.167.250)
Error: Failed to download metadata for repo 'baseos': Cannot prepare internal mirrorlist: Status code: 404 for https://mirrors.almalinux.org/mirrorlist/$releasever/baseos (IP: 52.200.132.63)

Running it with yum update --releasever=8.9 works fine, after I have disabled a few repos that complain.

I have compared these two servers with the working server and all repos and so, but they are all three configured EXACTLY the same.

cat /etc/redhat-release
AlmaLinux release 8.9 (Midnight Oncilla)

cat /etc/os-release
NAME="AlmaLinux"
VERSION="8.9 (Midnight Oncilla)"
ID="almalinux"
ID_LIKE="rhel centos fedora"
VERSION_ID="8.9"
PLATFORM_ID="platform:el8"
PRETTY_NAME="AlmaLinux 8.9 (Midnight Oncilla)"
ANSI_COLOR="0;34"
LOGO="fedora-logo-icon"
CPE_NAME="cpe:/o:almalinux:almalinux:8::baseos"
HOME_URL="https://almalinux.org/"
DOCUMENTATION_URL="https://wiki.almalinux.org/"
BUG_REPORT_URL="https://bugs.almalinux.org/"

ALMALINUX_MANTISBT_PROJECT="AlmaLinux-8"
ALMALINUX_MANTISBT_PROJECT_VERSION="8.9"
REDHAT_SUPPORT_PRODUCT="AlmaLinux"
REDHAT_SUPPORT_PRODUCT_VERSION="8.9"

I have tried following this: Error: Failed to download metadata for repo 'BaseOS': Cannot download repomd.xml: Cannot download repodata/repomd.xml:All mirrors were tried · Issue #123 · AlmaLinux/almalinux-deploy · GitHub but I don’t have files /etc/yum.repos.d/CentOS-Base.repo and /etc/yum.repos.d/CentOS-AppStream.repo and even if I did, this would just disable it, which is not ideally what we want to do in this case.

And as mentioned when we run yum update --releasever=8.9 some repos complain, which means we need to disable them, and incline they would never update accordingly.

I would appreciate any assistance in this regard, thank you.

gran061 · May 27, 2024, 8:41am

I forgot to mention that I have also commented the mirror URL and uncommented the correct baseurl in the almalinux.rep file and now it throws:

yum update
Unable to detect release version (use '--releasever' to specify release version)
AlmaLinux $releasever - BaseOS                                                                                                                                 94  B/s | 153  B     00:01    
Errors during downloading metadata for repository 'baseos':
  - Status code: 404 for https://repo.almalinux.org/almalinux/$releasever/BaseOS/x86_64/os/repodata/repomd.xml (IP: 67.222.0.50)
Error: Failed to download metadata for repo 'baseos': Cannot download repomd.xml: Cannot download repodata/repomd.xml: All mirrors were tried

This is happening for all [baseos], [appstream] and [extras] in the .repo file.

jlehtone · May 27, 2024, 9:00am

First, the releasever ought to be 8. That way it does not need to be changed (in systems) when a point update is released. The mirrorlist server simply starts to give updated URLs when we request for “8”. Likewise, the baseurl leads to “directory 8” on repo server, that is a symlink to latest version.

If repo does not actually have “8.9”, but only “real 8”, then the 8.9 will be an error.

This is how you see what dnf sees:

# /usr/libexec/platform-python -c 'import dnf, json; db = dnf.dnf.Base(); print(json.dumps(db.conf.substitutions, indent=2))'
{
  "arch": "x86_64",
  "basearch": "x86_64",
  "releasever": "8"
}

and

# dnf config-manager --dump-variables
arch = x86_64
basearch = x86_64
infra = stock
stream = 8-stream
contentdir = almalinux
releasever = 8

(The outputs are from AlmaLinux 8 system.)

How does it get the values? I don’t know. strace shows that dnf reads from /etc/os-release

Check that release files are ok with:

rpm -V almalinux-release

Is something missing or modified?

gran061 · May 27, 2024, 9:16am

Here’s the result:

rpm -V almalinux-release
package almalinux-release is not installed

But I ran rpm -V almalinux-release on a Working machine and on the non-working machine, they both returned the exact same.

I am not exactly sure what could be missing, I looked at both the working and non-working machines and both /etc/yum.repos.d/contents looks exactly the same.

gran061 · May 27, 2024, 9:19am

Sorry, wait I lied. When I run rpm -V almalinux-release on the Working machine, it returns nothing. Sorry, I got confused, because all three server names are similar.

So this is what the working machine shows:

root@hostname ~ $ rpm -V almalinux-release
root@hostname ~ $

And with the non-working machines I get this:

rpm -V almalinux-release
package almalinux-release is not installed

jlehtone · May 27, 2024, 9:20am

Then start with:
dnf --releasever=8 install almalinux-release

gran061 · May 27, 2024, 9:24am

Thank you very much, this works and solves it. But do you have any idea why would it just be missing? I just accessed the server to run a normal yum/dnf update like I do and this just randomly happened.

jlehtone · May 27, 2024, 9:39am

One can see which transactions have involved that package:

# dnf history list almalinux-release

The package setup is “protected” and requires ‘system-release’. Since almalinux-release provides ‘system-release’, it should not be possible to remove it (with dnf). Besides, a proper removal would have removed the repo definitions too. Perhaps the database about installed packages cot corrupted somehow?

gran061 · May 27, 2024, 9:52am

We have been struggling with constant RPM DB corruptions on these two machines but we typically remedy them by running rpm --rebuilddb. We don’t know what’s causing the constant RPM corruption and don’t have the necessary expertise to investigate deeply.

gran061 · May 31, 2024, 6:54pm

Hi @jlehtone I would like to pick this up again. It seems something else is still up. I am unable to update packages.

On the working machine, there were over 200 packages that needed to be updated today, but on these two servers, there was nothing. It says everything is updated. And I was confused so I tried to check the version for the SSH server.

On the working machine I get this:

root@hostname ~ $ rpm -qa | grep openssh
openssh-8.0p1-24.el8.x86_64
openssh-server-8.0p1-24.el8.x86_64
openssh-clients-8.0p1-24.el8.x86_64

root@hostname ~ $ dnf list installed openssh-server
Installed Packages
openssh-server.x86_64                                                                           8.0p1-24.el8                                                                           @baseos
root@hostname ~ $

On the previous non-working machines, I get this:

root@hostname /etc/yum.repos.d $ rpm -qa | grep openssh
root@hostname /etc/yum.repos.d $ dnf list installed openssh-server
Error: No matching Packages to list
root@hostname /etc/yum.repos.d $

It makes no sense, I am SSHed into these two servers as we speak, yet it says these packages do not exist… this most likely brings us back to corrupted rpm database, any ideas how I can fix this?

EDIT:

This then also explains why almalinux-release was magically “missing”, which means it is in fact installed but for some reason rpm did not pick it up, until I had to “install” it.