Recurring rpm database corruption

Support
1

Hi Guys

I manage multiple Almalinux 8 VPS and dedicated servers, and recently I have been experiencing a lot of RPM Database corruption specifically this one:

error: rpmdb: BDB0113 Thread/process 401048/139672503925632 failed: BDB1507 Thread died in Berkeley DB library
error: db5 error(-30973) from dbenv->failchk: BDB0087 DB_RUNRECOVERY: Fatal error, run database recovery
error: cannot open Packages index using db5 -  (-30973)
error: cannot open Packages database in /var/lib/rpm
Error: Error: rpmdb open failed

And yes, I have run this MANY times:

mkdir -p /tmp/rpm-backup
cp -vf /var/lib/rpm/__db.* /tmp/rpm-backup
rm -vf /var/lib/rpm/__db.*

rpm --quiet -qa
rpm --rebuilddb
dnf clean all

This does solve the issue, but temporarily, and I do mean temporarily. I have one specific instance, as soon as I run dnf update -y after this, it downloads packages and tries to update it fine and as it installs, it fails with rpmdb error.

This issue is also concerning why I experienced issues like this: Unable to detect release version

I do not know what to do, there’s an underlying issue. Servers were rebooted multiple times and I do not have the experience or expertise to find out why this keeps happening and the main issue, it seems to happen with many instances I have, and not just one.

I would appreciate any assistance in specifically how to debug this exactly and provide you guys more details for any sort of assistance.

2

For what it is worth, I implemented dcrpm on less severe servers to run every night at 11 PM, as explained here: linux - How can I track down why the rpm DB on my servers keeps getting corrupted? - Server Fault

And for severe DB corruptions that happen in seconds, I was advised to follow this: rpm.org - RPM Database Recovery

I followed the instructions from “BDB Backend” and stopped right before " If all fails" as explained in the article and now I am monitoring to see if rpmdb breaks again.

Though, still no real answer as to why this continuously seems to happen.

3

Sadly, this did not solve the issue, as it keeps breaking so I am back to square one.