Almalinux 8.10 Network Forwarding between Interfaces not Working

Support
1

I have several Almalinux 8.10 VMs with 2 network interfaces, eth0 with a public IP and eth1 with a private IP(all VMs sharing the same private subnet).

I have a use case for example to be able to reach server B public IP from server A private IP interface passing through server B private interface.

I have the necessary routing command set in place and net.ipv4.ip_forward=1 and net.ipv4.conf.all.rp_filter=0 set. The same configuration has worked on Centos 7 machines (This was applied on many projects I worked on) but not Almalinux 8.

When trying to test the connection using a ping command, I can see that the requests are being received by server B and it is responding (tcpdump on the private interface of server B showing the received echo requests and replies) however the traffic is never received back on server A.

The same scenario above happens for example when I have a server B with a VPN configured on to connect to a remote site so the traffic from server A that is routed through server B machine to reach the remote site has the same behavior where echo and reply requests are received on server B but not forwarded back to its origin on server A.

So the question here what is the possible difference in the networking side between Centos 7 and Almalinux 8 that is preventing this communication from happening?

2

First, the kernel of CentOS 7 had netfilter. The kernel of AlmaLinux 8 does have that too, but mostly superceded with nf_tables. While FirewallD might add rules in both, it used iptables in the former and nftables in the latter. A difference can be in those details.

(You can see all active rules with sudo nft list ruleset )

Lets say that the two public IPs are on the same subnet, W, and privates in subnet L.

The W is link-local, so the A could contact B by passing packets “to W.b” out of the eth0.
You’d need a static rule “W.b via L.b” for them to leave through eth1. Perhaps even make sure that they are tagged “from L.a”.

Ok, B receives a “to W.b” packet via eth1. A packet for localhost, for the “INPUT” – no “FORWARD” here. Who should get the reply? The L.a. Who are we, the sender of reply? Logically L.b, but we are replying to someone, who reached for W.b, not L.b.

What SRC and DST are in the actual packets and do both interfaces see some of them, or just the one (that should)?

That sounds messy.

3

The issue is not related to firewalld and nftables as the same behavior is happening even if firewalld is turned off.

Please find below the scenario I mentioned with IPs, routes and traces taken for clarity.

Server A:
Private IP: 10.0.171.124/16
Public IP: 50.56.96.54/32

Server B:
Private IP: 10.0.174.20/16
Public IP: 50.56.101.32/32

I have the below route added on server A:

50.56.101.32/32 10.0.174.20 255.255.255.255 UGH 0 0 0 eth1

When I ping 50.56.101.32(public IP of server B) from server A I get the below on tcpdump running on server B eth1 (note that tcpdump on eth0 does not show any packet although in the first place the ping is destined to its IP however the same behavior is present on centos 7)

sudo tcpdump -i eth1 host 10.0.171.124 -s0 -nn
dropped privs to tcpdump
tcpdump: verbose output suppressed, use -v or -vv for full protocol decode
listening on eth1, link-type EN10MB (Ethernet), capture size 262144 bytes
00:49:52.754321 IP 10.0.171.124 > 50.56.101.32: ICMP echo request, id 80, seq 1, length 64
00:49:52.754380 IP 50.56.101.32 > 10.0.171.124: ICMP echo reply, id 80, seq 1, length 64
00:49:53.784720 IP 10.0.171.124 > 50.56.101.32: ICMP echo request, id 80, seq 2, length 64
00:49:53.784746 IP 50.56.101.32 > 10.0.171.124: ICMP echo reply, id 80, seq 2, length 64
00:49:54.808711 IP 10.0.171.124 > 50.56.101.32: ICMP echo request, id 80, seq 3, length 64
00:49:54.808740 IP 50.56.101.32 > 10.0.171.124: ICMP echo reply, id 80, seq 3, length 64
00:49:55.832610 IP 10.0.171.124 > 50.56.101.32: ICMP echo request, id 80, seq 4, length 64
00:49:55.832654 IP 50.56.101.32 > 10.0.171.124: ICMP echo reply, id 80, seq 4, length 64

So as shown above, the eth1 of server B is replying back and it is on the same subnet as eth1 of server A so it should know where 10.0.171.124(private IP of server A) is but the below is only shown on the tcpdump of server A:

sudo tcpdump -i any host 10.0.171.124 -s0 -nn | grep -i icmp
dropped privs to tcpdump
tcpdump: verbose output suppressed, use -v or -vv for full protocol decode
listening on any, link-type LINUX_SLL (Linux cooked v1), capture size 262144 bytes
00:49:52.749433 IP 10.0.171.124 > 50.56.101.32: ICMP echo request, id 80, seq 1, length 64
00:49:53.780496 IP 10.0.171.124 > 50.56.101.32: ICMP echo request, id 80, seq 2, length 64
00:49:54.804492 IP 10.0.171.124 > 50.56.101.32: ICMP echo request, id 80, seq 3, length 64
00:49:55.828406 IP 10.0.171.124 > 50.56.101.32: ICMP echo request, id 80, seq 4, length 64

4

When RHEL has multiple IPs configured, only one is reachable from a remote network. Or why does RHEL ignore packets when the route for outbound traffic differs from the route of incoming traffic? - Red Hat Customer Portal does explain rp_filter (and suggests ‘2’, or policy-based routing). Curiously, that appeared already in el6, so does not explain difference between el7 and el8.

The nf_tables does more than just filtering; NAT rules can be in it too.

5

Hello Jukka,

Thank you for the reply.

I already tested setting net.ipv4.conf.all.rp_filter to values 0 and 2 but that didn’t solve the issue. Moreover, there are no packets dropped appearing in nstat -rsz | grep IPReversePathFilter or netstat -s | grep IPReversePathFilter

Also in my case server A is sending packets to server B on an IP that isn’t from the same subnet but server B is actually receiving the packets and replying. For server B, its replying from eth1 to eth1 of server A which are on the same local network plus there is a default routing rule on server B for the 10.0.0.0/16 subnet to pass through eth1.

As for nf_tables, good to know but when I turn off the firewalld, all rules in the nf_tables are set to accept with no natting rules.

6

Recap:

  • We see ping leave from A to 50.56.101.32
  • We see ping from 10.0.171.124 (A) arrive to B via eth1
  • We see reply leave from B via eth1 (to 10.0.171.124)
  • We do not see the reply to reach A

Who ate it? If none, then does our filter (tcpdum host and grep) discard it?

7

After more checking, the scenario I mentioned used to work up to Almalinux 8.8 and started failing in Almalinux 8.9 release.