Almalinux 9.4 Static Network Not Saving After Reboot

stevewest15 · August 4, 2024, 10:31pm

OS Version: AlmaLinux release 9.4 (Seafoam Ocelot)
uname: 5.14.0-427.28.1.el9_4.x86_64 #1 SMP PREEMPT_DYNAMIC

I have used Network Manager to configure two network cards (enX0 and enX1) which are both setup with static IP addresses. However after a reboot the server, the IP address assigned to each network get switched to the other network:

enX0 has a static IP address of 10.10.10. 67
enX1 has a static IP address of 192.168.1.30

But after a reboot, the IP addresses switch like so:

enX0 has a static IP address of 192.168.1.30
enX1 has a static IP address of 10.10.10. 67

Here is the output of the network before rebooting the server:

# nmcli device show
GENERAL.DEVICE:                         enX1
GENERAL.TYPE:                           ethernet
GENERAL.HWADDR:                         56:C9:D0:E9:E5:7A
GENERAL.MTU:                            1500
GENERAL.STATE:                          100 (connected)
GENERAL.CONNECTION:                     enX1
GENERAL.CON-PATH:                       /org/freedesktop/NetworkManager/ActiveConnection/5
WIRED-PROPERTIES.CARRIER:               on
IP4.ADDRESS[1]:                         192.168.1.30/24
IP4.GATEWAY:                            192.168.1.2
IP4.ROUTE[1]:                           dst = 192.168.1.0/24, nh = 0.0.0.0, mt = 102
IP4.ROUTE[2]:                           dst = 0.0.0.0/0, nh = 192.168.1.2, mt = 102
IP4.DNS[1]:                             8.8.8.8
IP4.DNS[2]:                             8.8.4.4
IP6.ADDRESS[1]:                         fe80::3fea:674d:bb72:21eb/64
IP6.GATEWAY:                            --
IP6.ROUTE[1]:                           dst = fe80::/64, nh = ::, mt = 1024

GENERAL.DEVICE:                         enX0
GENERAL.TYPE:                           ethernet
GENERAL.HWADDR:                         82:5A:D2:E1:CC:A9
GENERAL.MTU:                            1500
GENERAL.STATE:                          100 (connected)
GENERAL.CONNECTION:                     enX0
GENERAL.CON-PATH:                       /org/freedesktop/NetworkManager/ActiveConnection/6
WIRED-PROPERTIES.CARRIER:               on
IP4.ADDRESS[1]:                         10.10.10.67/24
IP4.GATEWAY:                            --
IP4.ROUTE[1]:                           dst = 10.10.10.0/24, nh = 0.0.0.0, mt = 103
IP6.ADDRESS[1]:                         fe80::805a:d2ff:fee1:cca9/64
IP6.GATEWAY:                            --
IP6.ROUTE[1]:                           dst = fe80::/64, nh = ::, mt = 1024

GENERAL.DEVICE:                         lo

And here is the network after rebooting the server: (the IP addresses get switched to the wrong device where enX0 gets IP address of enX1 and enX1 gets IP address of enX0)

GENERAL.DEVICE: enX0
GENERAL.TYPE: ethernet
GENERAL.HWADDR: 82:5A:D2:E1:CC:A9
GENERAL.MTU: 1500
GENERAL.STATE: 100 (connected)
GENERAL.CONNECTION: System enX1:1
GENERAL.CON-PATH: /org/freedesktop/NetworkManager/ActiveConnection/4
WIRED-PROPERTIES.CARRIER: on
IP4.ADDRESS[1]: 192.168.1.30/24
IP4.GATEWAY: --
IP4.ROUTE[1]: dst = 192.168.1.0/24, nh = 0.0.0.0, mt = 101
IP6.ADDRESS[1]: fe80::805a:d2ff:fee1:cca9/64
IP6.GATEWAY: --
IP6.ROUTE[1]: dst = fe80::/64, nh = ::, mt = 256

GENERAL.DEVICE: enX1
GENERAL.TYPE: ethernet
GENERAL.HWADDR: 56:C9:D0:E9:E5:7A
GENERAL.MTU: 1500
GENERAL.STATE: 100 (connected)
GENERAL.CONNECTION: System enX0:1
GENERAL.CON-PATH: /org/freedesktop/NetworkManager/ActiveConnection/3
WIRED-PROPERTIES.CARRIER: on
IP4.ADDRESS[1]: 10.10.10.67/24
IP4.GATEWAY: --
IP4.ROUTE[1]: dst = 10.10.10.0/24, nh = 0.0.0.0, mt = 100
IP6.ADDRESS[1]: fe80::54c9:d0ff:fee9:e57a/64
IP6.GATEWAY: --
IP6.ROUTE[1]: dst = fe80::/64, nh = ::, mt = 256

GENERAL.DEVICE: lo
GENERAL.TYPE: loopback
GENERAL.HWADDR: 00:00:00:00:00:00
GENERAL.MTU: 65536
GENERAL.STATE: 100 (connected (externally))
GENERAL.CONNECTION: lo
GENERAL.CON-PATH: /org/freedesktop/NetworkManager/ActiveConnection/1
IP4.ADDRESS[1]: 127.0.0.1/8
IP4.GATEWAY: --
IP6.ADDRESS[1]: ::1/128
IP6.GATEWAY: --
IP6.ROUTE[1]: dst = ::1/128, nh = ::, mt = 256

Any suggestions on what I can check that would be causing this issue?

Thank you,

SW

jlehtone · August 5, 2024, 6:49am

What do you get with:

nmcli c s

stevewest15 · August 5, 2024, 6:37pm

@jlehtone Thank you for your assistance! Here is the output:

# nmcli c s
NAME  UUID                                  TYPE      DEVICE
enX1  dae0559d-49dd-70b6-d090-8540ba98128e  ethernet  enX0
enX0  7bdc3ea6-b6dc-674a-a45a-fac8d2defe31  ethernet  enX1
lo    8776d047-8903-4ebb-8191-70e27c1ae301  loopback  lo

So I rebuild the network so that enX0 matches enX0 and vise versa:

NAME  UUID                                  TYPE      DEVICE
enX1  dae0559d-49dd-70b6-d090-8540ba98128e  ethernet  enX1
enX0  7bdc3ea6-b6dc-674a-a45a-fac8d2defe31  ethernet  enX0
lo    8776d047-8903-4ebb-8191-70e27c1ae301  loopback  lo

And then restarted the server and that appears to have fixed my issue. Not sure how I got the name and device id mixed up.

Thank you again for your help!

SW

jlehtone · August 6, 2024, 8:18am

IMHO, that does not quite explain the issue. Names of connections should not matter.

Do look at:

nmcli -f connection.interface-name,802-3-ethernet.mac-address c s enX0
nmcli -f connection.interface-name,802-3-ethernet.mac-address c s enX1

The NetworkManager does use one of those fields to decide which device the connection is used for.
When you did install Alma, its installer did define connections, and did set the 802-3-ethernet.mac-address
When you define connection with nmcli, you must specify ‘ifname’, the connection.interface-name

One can change that with:

nmcli con mod enX0 802-3-ethernet.mac-address 82:5A:D2:E1:CC:A9 connection.interface-name ""

This makes the connection “enX0” to be used for device that has MAC address 82:5A:D2:E1:CC:A9.

The thing is that device names can change, but the MAC should always point at same physical device.

When machine boots (or if network card is plugged in) the NetworkManager checks all defined connections, whether any can be used for device. (Repeats for each device.) There can be many connections defined for device, but only one is used. If no stored connection is for device “eno1”, then NetworkManager creates connection “System eno1” for it. This is not stored, so will not survive reboot. (This auto-generation can be disabled by installing package NetworkManager-config-server.)

Since the “wrong” connections seemed to have name of connection like “System …” and you had no such stored connections, it looks like the NetworkManager did fail to match existing definitions. However, the autogenerated connections do use DHCP, so should not have gotten the static addresses.

Fascinating, or strange.

stevewest15 · August 6, 2024, 10:55am

Hi @jlehtone ,

Thank you for your assistance! I was mistaken that my issue was resolved. I rebooted the server and ran into the same issue again.

I’m new to almalinux as I moved from Centos 7.x. I think I’m doing something wrong with how I’m configuring the network.

Just a couple of questions:

Does almalinux auto-populate the mac address from the network card or is that something that I need to manually enter for each interface?
Does almalinux read the files under /etc/NetworkManager/system-connections/ directory at each boot or is the network settings stored elsewhere?

I checked several of the almalinux servers none of them show a mac address when running the following commands:

# nmcli -f connection.interface-name,802-3-ethernet.mac-address c s enX0
connection.interface-name:              enX0
802-3-ethernet.mac-address:             --
# nmcli -f connection.interface-name,802-3-ethernet.mac-address c s enX1
connection.interface-name:              enX1
802-3-ethernet.mac-address:             --

However, the files under /etc/NetworkManager/system-connections/ do have the correct MAC address for each interface.

I did follow your instructions and run the following commands to manually assign mac address to each connection interface:

# nmcli con mod enX0 802-3-ethernet.mac-address 8e:ad:83:f9:32:60 connection.interface-name "enX0"
# nmcli -f connection.interface-name,802-3-ethernet.mac-address c s enX0
connection.interface-name:              enX0
802-3-ethernet.mac-address:             8E:AD:83:F9:32:60

# nmcli con mod enX1 802-3-ethernet.mac-address 06:e5:12:84:42:d8 connection.interface-name "enX1"
# nmcli -f connection.interface-name,802-3-ethernet.mac-address c s enX1
connection.interface-name:              enX1
802-3-ethernet.mac-address:             06:E5:12:84:42:D8

And lastly, am I missing some critical step when configuring the server after a fresh install? Here are the commands I’m running to configure the network:

# nmcli device status
DEVICE    TYPE                STATE                                   CONNECTION
enX0        ethernet        connected                            enX0
lo              loopback       connected (externally)       lo
enX1        ethernet        disconnected                        Wired connection 1

# nmcli connection modify "Wired connection 1" connection.id "enX1"

# nmcli device status
DEVICE  TYPE      STATE                                  CONNECTION
enX0    ethernet  connected                              enX0
enX1    ethernet  connecting (getting IP configuration)  enX1
lo      loopback  connected (externally)                 lo

# nmcli con mod enX0 ipv4.addresses 10.10.10.50 ipv4.method manual

# nmcli con mod enX1 ipv4.addresses "192.168.1.50/24,192.168.1.55" ipv4.gateway 192.168.1.1 ipv4.dns "8.8.8.8,8.8.4.4" ipv4.method manual

# nmcli con up enX0

# nmcli con up enX1

# nmcli device show
GENERAL.DEVICE:                         enX1
GENERAL.TYPE:                           ethernet
GENERAL.HWADDR:                         06:E5:12:84:42:D8
GENERAL.MTU:                            1500
GENERAL.STATE:                          100 (connected)
GENERAL.CONNECTION:                     enX1
GENERAL.CON-PATH:                       /org/freedesktop/NetworkManager/ActiveConnection/4
WIRED-PROPERTIES.CARRIER:               on
IP4.ADDRESS[1]:                         192.168.1.55/24
IP4.ADDRESS[2]:                         192.168.1.50/24
IP4.GATEWAY:                            192.168.1.1
IP4.ROUTE[1]:                           dst = 192.168.1.0/24, nh = 0.0.0.0, mt = 102
IP4.ROUTE[2]:                           dst = 192.168.1.0/24, nh = 0.0.0.0, mt = 102
IP4.ROUTE[3]:                           dst = 0.0.0.0/0, nh = 192.168.1.1, mt = 102
IP4.DNS[1]:                             8.8.8.8
IP4.DNS[2]:                             8.8.4.4
IP6.ADDRESS[1]:                         fe80::6b7b:791e:5f97:ecd0/64
IP6.GATEWAY:                            --
IP6.ROUTE[1]:                           dst = fe80::/64, nh = ::, mt = 1024

GENERAL.DEVICE:                         enX0
GENERAL.TYPE:                           ethernet
GENERAL.HWADDR:                         8E:AD:83:F9:32:60
GENERAL.MTU:                            1500
GENERAL.STATE:                          100 (connected)
GENERAL.CONNECTION:                     enX0
GENERAL.CON-PATH:                       /org/freedesktop/NetworkManager/ActiveConnection/2
WIRED-PROPERTIES.CARRIER:               on
IP4.ADDRESS[1]:                         10.10.10.50/24
IP4.GATEWAY:                            --
IP4.ROUTE[1]:                           dst = 10.10.10.0/24, nh = 0.0.0.0, mt = 101
IP6.ADDRESS[1]:                         fe80::8cad:83ff:fef9:3260/64
IP6.GATEWAY:                            --
IP6.ROUTE[1]:                           dst = fe80::/64, nh = ::, mt = 1024

GENERAL.DEVICE:                         lo
GENERAL.TYPE:                           loopback
GENERAL.HWADDR:                         00:00:00:00:00:00

And here are the contents of files under /etc/NetworkManager/system-connections/ directory:

# ls -al /etc/NetworkManager/system-connections/
total 8
drwxr-xr-x. 2 root root  56 Jul 29 16:28 .
drwxr-xr-x. 7 root root 134 Jul 24 04:23 ..
-rw-------. 1 root root 289 Jul 29 16:28 enX0.nmconnection
-rw-------. 1 root root 331 Jul 18 00:37 enX1.nmconnection

# cat /etc/NetworkManager/system-connections/enX0.nmconnection
[connection]
id=enX0
uuid=650a3f60-3795-384b-bd38-6893baa7164d
type=ethernet
autoconnect-priority=-999
interface-name=enX0
timestamp=1722935935

[ethernet]
mac-address=8E:AD:83:F9:32:60

[ipv4]
address1=10.10.10.50/24
method=manual
never-default=true

[ipv6]
addr-gen-mode=eui64
method=auto
never-default=true

[proxy]

# cat /etc/NetworkManager/system-connections/enX1.nmconnection
[connection]
id=enX1
uuid=d8a8642d-f8ca-3239-8b11-b42ed59ac441
type=ethernet
autoconnect-priority=-999
interface-name=enX1
timestamp=1722940154

[ethernet]
mac-address=06:E5:12:84:42:D8

[ipv4]
address1=209.8.232.50/24,209.8.232.1
address2=209.8.232.55/24
dns=8.8.8.8;8.8.4.4;
method=manual

[ipv6]
addr-gen-mode=default
method=auto
never-default=true

[proxy]

Please let me know if I’m missing some steps on how to configure the network. I only use static IP addresses so this is why I’m issuing “ipv4.method manual”.

Thank you once again!

SW

jlehtone · August 6, 2024, 12:35pm

CentOS 7 did have NetworkManager too. It (NM) did start to support most of my needs on the 7.3. I have hence learned to use it already way before AlmaLinux. Granted, the NM in AlmaLinux does support more features and special cases than the version in CentOS 7.

I might have mistaken. Oh, well …

Mine don’t. If not in 802-3-ethernet.mac-address, then is should not be in the file either.

I prefer:

connection.interface-name:              --
802-3-ethernet.mac-address:             8E:AD:83:F9:32:60

as then there is exactly one device that this connection is for.

Each hardware device should have a MAC address in it. That is used by default. If you want to store it in the config, then you have to “manually” set it. I have DHCP server(s) and they are set to give addresses to known MACs only, so I have to look up the MAC of each system anyway. (Only for known machines, and same IP every time. “Static”, but via DHCP.)

It is possible to override that in software. For example, WiFi on laptops may use randomly generated MAC, so “internet cafe’s” can’t show with their logs that you were there.

Up to el8 the NetworkManager did read and write in the legacy “initscripts format” to /etc/sysconfig/network-scripts/, except some connection types that were written to /etc/NetworkManager/system-connections/
In el9 all connections are stored in /etc/NetworkManager/system-connections/, but the NM might still read /etc/sysconfig/network-scripts/ifcfg-*.

Note though that a user is apparently able to create “personal” connections – e.g. for WiFi – and those are probably not stored under /etc (because users cannot write there).

Did you have a typo there? No prefix. (You ‘device show’ shows “24”.)

I have started to use Ansible for managing my systems. There are packages ‘ansible-core’ and ‘rhel-system-roles’ (and EPEL has additional roles/modules in package ansible).
See Chapter 19. Configuring network settings by using the RHEL system role | Red Hat Product Documentation

stevewest15 · August 6, 2024, 7:55pm

@jlehtone Thank you so much for all of the very helpful information!

Thank you for the tip on not setting “connection.interface-name” when setting mac address:

# nmcli con mod enX0 802-3-ethernet.mac-address D3:D3:D3:D3:D3:D3 connection.interface-name ""
# nmcli -f connection.interface-name,802-3-ethernet.mac-address c s enX0
connection.interface-name:              --
802-3-ethernet.mac-address:             D3:D3:D3:D3:D3:D3

In my use case, I need to set static IP addresses (private and public) to one or more network cards on each server:

# nmcli device status

DEVICE      TYPE          STATE                                    CONNECTION
enX1        ethernet      connected                                enX1
enX0        ethernet      connected                                enX0
enX2        ethernet      connecting (getting IP configuration)    enX2
lo          loopback      connected (externally)                   lo

Can I just create or edit the files in /etc/NetworkManager/system-connections/ and then have NetworkManager read those files to bring up the network?

It just seems like not all network information is stored in the various files under /etc/NetworkManager/system-connections/enX0.nmconnection. For example, NetworkManager doesn’t store in /etc/NetworkManager/system-connections/enX0.nmconnection if I change the “connection.interface-name” when setting the mac address:

# nmcli con mod enX0 802-3-ethernet.mac-address D3:D3:D3:D3:D3:D3 connection.interface-name ""
[root@goofy-chebyshev ~]# nmcli -f connection.interface-name,802-3-ethernet.mac-address c s enX0
connection.interface-name:              --
802-3-ethernet.mac-address:             D3:D3:D3:D3:D3:D3
[root@goofy-chebyshev ~]# cat /etc/NetworkManager/system-connections/enX0.nmconnection
[connection]
id=enX0
uuid=650a3f60-3795-384b-bd38-6893baa7164d
type=ethernet
autoconnect-priority=-999
timestamp=1722971644

[ethernet]
mac-address=D3:D3:D3:D3:D3:D3

[ipv4]
address1=10.10.10.113/24
method=manual
never-default=true

[ipv6]
addr-gen-mode=eui64
method=auto

[proxy]
[root@goofy-chebyshev ~]# nmcli con mod enX0 802-3-ethernet.mac-address D3:D3:D3:D3:D3:D3 connection.interface-name "enX0"
[root@goofy-chebyshev ~]# cat /etc/NetworkManager/system-connections/enX0.nmconnection
[connection]
id=enX0
uuid=650a3f60-3795-384b-bd38-6893baa7164d
type=ethernet
autoconnect-priority=-999
interface-name=enX0
timestamp=1722971644

[ethernet]
mac-address=D3:D3:D3:D3:D3:D3

[ipv4]
address1=10.10.10.113/24
method=manual
never-default=true

[ipv6]
addr-gen-mode=eui64
method=auto

[proxy]

Not sure if that is critical or not, just looking for best practices to ensure network settings survive after each server reboot. I prefer not to complicate it (thanks for the tip re Ansible) but coming from centos 7, I prefer to update configuration files if NetworkManager reads those files under /etc/NetworkManager/system-connections/after each server reboot or when a network device is added/removed.

Thank you again!

SW

jlehtone · August 6, 2024, 9:45pm

Looks like that like was added …

Edit a file
Use nmcli, nmtui, … (i.e. NetworkManager) to modify the file(s)
Use Ansible to tell NetworkManager to modify the file(s)

The rationale of not editing the file directly is that manual edits can introduce typos and syntax errors. The latter tools ensure that the syntax is correct and config is more coherent. Alas, nothing protects from logical errors.

The benefit that I see in Ansible is that I have a copy of config of multiple machines in one place, “inventory” (as YAML files), and I can re-deploy whenever needed. I can reinstall machine (packages and config) with minimal effort. Furthermore, I have the inventory in Git repo and have clone of it elsewhere: backup of config of multiple machines. Yes, setting that up does require initial investment. IMHO, it was worth it for my needs.

Like I said, I do set “static IP addresses” too, with DHCP. Only the machine that runs DHCP server (dnsmasq in my systems) for a subnet does require manual address on that subnet. If you have no control over DHCP server – can’t even ask for static addresses – then manual is necessary. (Some feel that manual config has less “overhead”. True, when it is correct and does not need to change.)

Btw, you don’t seem to set IPv6 addresses. If you don’t use IPv6 (yet), then you can add ipv6.method disabled for your connections.