Containers in Almalinux have no network access

Support
1

I am uncertain where to post this so I am posting it here. If I should post it elsewhere, please let me know.

I think I found a bug in Almalinux 9.1, involving network access for containers.

I am running docker on a machine running Almalinux 9.1, and I am running a container that is required to use the Host network. When I attempt to access this application (urbackup) through its web interface, I keep getting “connection refused” errors.

Doing a netstat on my Almalinux host, I am seeing that for some reason the ports that the docker container should be listening on are not visible. In fact, when I do a netstat looking for those ports, I get the following:

$ sudo netstat -anp | grep LISTEN | grep urbackup
$

this indicates to me that the host is not seeing the ports exposed by the container.

It should be noted that I have other dockerized applications running without problems. None of these other applications run in host mode.

I did try putting this application on one of my other Almaliux machines using docker. I get the same behavior.

I also deliberately went to dockerhub and found another container that requires host mode. ran it on docker and it did not run properly. It failed due to network connection problems. Host mode apparently does not work on Almaliux.

The urbackup docker container works without problems on Ubuntu and other Linux operating systems.

I also run LXD on one of my machines. I created an Ubuntu container using lxc, went into its shell, and attempted to do a standard update. I got the following:

~# sudo apt update
Ign:1 http://archive.ubuntu.com/ubuntu jammy InRelease
Ign:2 http://security.ubuntu.com/ubuntu jammy-security InRelease
Ign:3 http://archive.ubuntu.com/ubuntu jammy-updates InRelease
Ign:2 http://security.ubuntu.com/ubuntu jammy-security InRelease
Ign:2 http://security.ubuntu.com/ubuntu jammy-security InRelease
Ign:4 http://archive.ubuntu.com/ubuntu jammy-backports InRelease
Err:2 http://security.ubuntu.com/ubuntu jammy-security InRelease
  Temporary failure resolving 'security.ubuntu.com'
Ign:1 http://archive.ubuntu.com/ubuntu jammy InRelease
Ign:3 http://archive.ubuntu.com/ubuntu jammy-updates InRelease
Ign:4 http://archive.ubuntu.com/ubuntu jammy-backports InRelease
Ign:1 http://archive.ubuntu.com/ubuntu jammy InRelease
Ign:3 http://archive.ubuntu.com/ubuntu jammy-updates InRelease
Ign:4 http://archive.ubuntu.com/ubuntu jammy-backports InRelease
Err:1 http://archive.ubuntu.com/ubuntu jammy InRelease
  Temporary failure resolving 'archive.ubuntu.com'
Err:3 http://archive.ubuntu.com/ubuntu jammy-updates InRelease
  Temporary failure resolving 'archive.ubuntu.com'
Err:4 http://archive.ubuntu.com/ubuntu jammy-backports InRelease
  Temporary failure resolving 'archive.ubuntu.com'
Reading package lists... Done
Building dependency tree... Done
Reading state information... Done
All packages are up to date.
W: Failed to fetch http://archive.ubuntu.com/ubuntu/dists/jammy/InRelease  Temporary failure resolving 'archive.ubuntu.com'
W: Failed to fetch http://archive.ubuntu.com/ubuntu/dists/jammy-updates/InRelease  Temporary failure resolving 'archive.ubuntu.com'
W: Failed to fetch http://archive.ubuntu.com/ubuntu/dists/jammy-backports/InRelease  Temporary failure resolving 'archive.ubuntu.com'
W: Failed to fetch http://security.ubuntu.com/ubuntu/dists/jammy-security/InRelease  Temporary failure resolving 'security.ubuntu.com'
W: Some index files failed to download. They have been ignored, or old ones used instead.

In fact, all attempts to perform network operations on my LXD container fail. The same operations do not fail if the LXD container is created on an Ubuntu host.

Almalinux apparently is failing to support certain network drivers that other Linux distros do. Is this a bug in Almalinux? Or is there some network configuration change that I need to do in order to make these things work?

Someome please advise.

2

guessing mode…firewalld configuration?

3

CanadaGuy:

firewalld isn’t the problem (though I originally thought it might be).

The necessary ports are open on the machine, and other docker containers (those not running in Host mode) can be accessed because each time I put one on I open any ports it listens on. I can see those ports with netstat. I don’t see the ports used by any container in Host mode.

Unless there is a special configuration of firewalld required for containers running in Host mode or VMs using bridge networks, firewalld is not the problem.

4

I have found a working “solution” to this issue:

Do not use Almalinux for running dockerized applications in host mode.

First, I investigated issues raised about containers that have network difficulties on Almalinux. There were some issues that were resolved (by opening ports on firewalld, correcting bad port settings in their docker-compose.yml files, etc), but all of the unresolved issues involved containers running in Host mode.

In order to determine if the problem was, indeed, caused by a bug in Almalinux, I installed my urbackup container on an Ubuntu system that I happened to have set up for another purpose. I was able to access it on Ubuntu with no problems.

Update note to CanadaGuy: I made sure to activate Ubuntu’s firewall prior to installing the urbackup container. I opened the ports that urbackup listened on, then started the container. Everything worked. I then closed the ports and attempted access to the container. As expected, I was unable to access it.

It is clear (at least to me) that Almalinux has a bug in whatever host driver its version of docker uses. Host mode simply does not work on Almalinux. The only thing I do not know (that maybe someone will find out) is if this bug is specifically on Almalinux or if it is common to all other CentOS-derived distributions.

And it isn’t just a problem with docker in host mode. I have also attempted to use LXD on Almalinux (one time to run an Ubuntu-based virtual container for some other tests). I experienced similar network connection/access problems as those seen with host mode docker containers. Apparently whatever bug is in Almalinux’s container driver affects all containerized network access.

Unfortunately, since no one seems interested in this issue (as indicated by the dearth of discussions about it in this thread as well as the incredible lack of information about similar issues on the Internet) it is unlikely that this bug will even be investigated, much less fixed. This means that, at least for the foreseeable future, Almalinux is unsuitable for running any docker containers that run in Host mode or any LXD containers. The good news is that other containers on docker, the ones where you don’t run in Host mode, seem to work without problems – so far.

Unfortunately, I have a number of containers that I will need to run in that mode, including certain security applications and network monitoring applications. I am also very impressed with LXD and plan to run a number of virtual OSs in LXD containers. In order to run these things, I may have to replace several of my Almalinux installations (bare metal and virtual) with Ubuntu.

I am posting this solution in order to inform anyone else who runs into similar network access problems on their containers. My message to you is that if you are running those containers on Linux and if those containers are running in host mode or using LXD, check your distribution. The problem may be with the virtual network driver, especially if you are using a CentOS-derived distro. Your problem will certainly be the distro if you are running them on Almalinux.