KVM shows network bridge error

General Discussion & Chat
1

Hi everybody,

I am a newbie in Linux.
I have an issue about the network access in a VM. When I tried to PowerON the VM. It returns error,

Error starting domain: Unable to add bridge ens224.242 port vnet9: Operation not supported

Maybe I misconfigure the network bridge/VLAN settings in KVM/VM … because this is my first time to do it

Here is the background

  • Almalinux9
  • KVM
    • NIC1: ens192 (10.10.10.74, management purpose)
    • NIC2: ens224 (trunk for the VM)
  • VM
    • NIC1: 192.168.119.74 (assigned vlan id 242)
  • physical switch port setting is trunk mode, allow all VLANs

connection path
Switch port — (trunk) — [ens224]Linux — KVM — VM

below is the Virtual Network interface setting in VM

  • Network source: Bridge device…
  • Device name: ens242.224
    Device model: virtio
    IP address: Unknown
    Link state: active

below is the ifcfg

/etc/sysconfig/network-scripts/ifcfg-br0
DEVICE=br0
TYPE=Bridge
NAME=br0
BOOTPROTO=none
ONBOOT=yes
DELAY=0

/etc/sysconfig/network-scripts/ifcfg-ens224
DEVICE=ens224
NAME=ens224
TYPE=Ethernet
BOOTPROTO=none
ONBOOT=yes
BRIDGE=br0

/etc/sysconfig/network-scripts/ifcfg-ens224.242
VLAN=yes
DEVICE=ens224.242
NAME=ens224.242
BOOTPROTO=none
ONBOOT=yes
TYPE=Ethernet
BRIDGE=br0

Is there anybody could help please?

Thanks in advance

2

First, network configuration is managed with NetworkManager (NM) service in AlmaLinux.
Expectation is to use NM tools rather than touching/looking at config files directly.
In el9 NM no longer uses legacy config format in ( etc/sysconfig/network-scripts/) by default.

There is a (yet more abstract) way to set the config (than the NM tools): Ansible System Roles.
See Chapter 20. Configuring network settings by using RHEL system roles | Red Hat Product Documentation

One could create a play “host-net.yml”, install Ansible:

dnf install ansible-core rhel-system-roles

and apply the play:

ansible-playbook host-net.yml

I’ll get back to that below.

There are more than one way to set the network. This is what I would do:

  • The physical wire outside of the baremetal host and its NIC2 (“ens224”) see tagged traffic of one or more VLAN
  • VLAN interface “vlan-242” picks from NIC2 only the incoming packets that have tag “242” and sends out packets tagged with “242”
  • Bridge “bridge-242” has the “vlan-242” as one of its “ports”. Packets on the bridge are untagged
  • VM guest is linked to the “bridge-242”, and does not know anything about VLANs

The “host-net.yml”:

- name: Configure the network
  hosts: localhost
  tasks:
  - name: Ethernet connection profile for bridge on VLAN 242
    ansible.builtin.include_role:
      name: rhel-system-roles.network
    vars:
      network_connections:
      - name: ens224
        type: ethernet
        autoconnect: yes
        mac: A0:B1:C2:D3:E4:F5
        ip:
          dhcp4: no
          ipv6_disabled: true
        ethtool:
          features: {}
        state: up
      - name: bridge-242
        interface_name: br242
        type: bridge
        ip:
          dhcp4: no
          ipv6_disabled: true
        ethtool:
          features: {}
        state: up
      - name: vlan-242
        interface_name: vlan-242
        type: vlan
        parent: ens224
        vlan:
          id: 242
        controller: bridge-242
        port_type:  bridge
        state: up

The A0:B1:C2:D3:E4:F5 should be the MAC address of the NIC2. The NM will use this connection (“ens224”) on the NIC that has that MAC.

The “vlan-242” has parent “ens224” and is a port of bridge “bridge-242”.

There is no direct connection between ens224 and the bridge; they don’t know of each other. Neither has IP address either.

The KVM should see the “br242” as something that the guest can use.

The “config file” of the VLAN will look something like:

# cat /etc/NetworkManager/system-connections/vlan-242.nmconnection
[connection]
id=vlan-242
uuid=...
type=vlan
interface-name=vlan-242
master=...
slave-type=bridge
timestamp=...

[ethernet]

[vlan]
flags=1
id=242
parent=ens224

[bridge-port]
1 Like
3

@jlehtone thanks for the feedback.
After proceed your suggestion, the VM is still unable to access the network.
below is the link and connection state
++++++

nmcli connection
NAME        UUID                                  TYPE      DEVICE   
ens192      736ddabf-56c1-35ac-b172-8bad800bf804  ethernet  ens192   
bridge-242  f13f77b1-15af-4c82-8703-6b0664472f08  bridge    br242    
ens224      ea92e3b1-be1c-43a1-a080-773a2dd1e0ea  ethernet  ens224   
vlan-242    d43fa5cc-0ec4-4f6a-9620-c6c9119361a0  vlan      vlan-242 
lo          32a84e22-dfa0-40f8-8f20-5ddd5860e7bf  loopback  lo       
virbr0      0bb6be61-8d90-4336-9134-77e6e3c62599  bridge    virbr0   
vnet0       bc07c805-9eba-4260-9343-95a9a0765fae  tun       vnet0    
Profile 1   dda38c1a-c134-47cd-baa5-08877c6ad653  ethernet  --       


ip link
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN mode DEFAULT group default qlen 1000
    link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
2: ens192: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc mq state UP mode DEFAULT group default qlen 1000
    link/ether 00:50:56:ad:c7:50 brd ff:ff:ff:ff:ff:ff
    altname enp11s0
3: ens224: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc mq state UP mode DEFAULT group default qlen 1000
    link/ether 00:50:56:ad:76:ad brd ff:ff:ff:ff:ff:ff
    altname enp19s0
4: vlan-242@ens224: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue master br242 state UP mode DEFAULT group default qlen 1000
    link/ether 00:50:56:ad:76:ad brd ff:ff:ff:ff:ff:ff
5: br242: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP mode DEFAULT group default qlen 1000
    link/ether 00:50:56:ad:76:ad brd ff:ff:ff:ff:ff:ff
6: virbr0: <NO-CARRIER,BROADCAST,MULTICAST,UP> mtu 1500 qdisc noqueue state DOWN mode DEFAULT group default qlen 1000
    link/ether 52:54:00:1c:0f:cd brd ff:ff:ff:ff:ff:ff
7: vnet0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue master br242 state UNKNOWN mode DEFAULT group default qlen 1000
    link/ether fe:54:00:c8:d3:05 brd ff:ff:ff:ff:ff:ff

++++++

When i was applying the host-net.yml, an error was found

fatal error: configuration error: connections[0].ethtool: invalid key ‘state’“, “stderr”: “\n”, “stderr_lines”: [”"]}

Then, I tried move the " state: up " above the “ethtool” in host-net.yml
After that, there was no error by running “ansible-playbook host-net.yml”

However, the VM is still unable to reach the network…

In addition, i found that there is no ens224 in “brctl show” …

$ brctl show
bridge name     bridge id               STP enabled     interfaces
br242           8000.005056ad76ad       no              vlan-242
                                                        vnet0
virbr0          8000.5254001c0fcd       yes

I am still looking for how to verify and debug

is anybody could help ?
Thanks in advance

4

First, neither el8 nor el9 has ‘brctl’ any more. There is command bridge in iproute package.
bridge link should tell most of what the older tool did.
(I guess you have installed bridge-utils from EPEL to get the brctl.)

Second, the state is as it should. Your bridge has two ports to the vlan and the VM.
We do not want to see the traffic of other VLANs on the bridge.

Plain nmcli shows a summary too.

On the host, do tcpdump -nn -i br242
Does that show any traffic?

If not, then tcpdump -nn -i ens224
Does that show any traffic?

If traffic is seen on both, then question is why the VM does not see any?

Example from virsh dumpxml myvm:

    <interface type='bridge'>
      <mac address='52:54:00:01:02:03'/>
      <source bridge='brxxx'/>
      <target dev='vnet0'/>
      <model type='virtio'/>
      <alias name='net0'/>
      <address type='pci' domain='0x0000' bus='0x01' slot='0x00' function='0x0'/>
    </interface>
5

Thanks for the feedback @jlehtone
Tried your suggestion and found an interesting result…

Below is bridge link and nmcli result

$ bridge link
3: ens224: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 master br242 state forwarding priority 32 cost 2
4: vlan-242@ens224: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 master br242 state forwarding priority 32 cost 100
7: vnet0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 master br242 state forwarding priority 32 cost 100
9: vnet2: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 master br242 state forwarding priority 32 cost 100

$ nmcli
ens192: connected to ens192
        "VMware VMXNET3"
        ethernet (vmxnet3), 00:50:56:AD:C7:50, hw, mtu 1500
        ip4 default
        inet4 10.10.10.74/24
        route4 10.10.10.0/24 metric 100
        route4 default via 10.10.10.1 metric 100
        inet6 fe80::250:56ff:fead:c750/64
        route6 fe80::/64 metric 1024

br242: connected to bridge-242
        "br242"
        bridge, 00:50:56:AD:76:AD, sw, mtu 1500

ens224: connected to ens224
        "VMware VMXNET3"
        ethernet (vmxnet3), 00:50:56:AD:76:AD, hw, mtu 1500

vlan-242: connected to vlan-242
        "vlan-242"
        vlan, 00:50:56:AD:76:AD, sw, mtu 1500
        controller br242

lo: connected (externally) to lo
        "lo"
        loopback (unknown), 00:00:00:00:00:00, sw, mtu 65536
        inet4 127.0.0.1/8
        inet6 ::1/128
        route6 ::1/128 metric 256

virbr0: connected (externally) to virbr0
        "virbr0"
        bridge, 52:54:00:1C:0F:CD, sw, mtu 1500
        inet4 192.168.122.1/24
        route4 192.168.122.0/24 metric 0

vnet0: connected (externally) to vnet0
        "vnet0"
        tun, FE:54:00:C8:D3:05, sw, mtu 1500
        controller br242
        inet6 fe80::fc54:ff:fec8:d305/64
        route6 fe80::/64 metric 256

vnet2: connected (externally) to vnet2
        "vnet2"
        tun, FE:54:00:23:BA:76, sw, mtu 1500
        controller br242
        inet6 fe80::fc54:ff:fe23:ba76/64
        route6 fe80::/64 metric 256

DNS configuration:
        servers: 10.10.10.170
        domains: ml.local
        interface: ens192

Use "nmcli device show" to get complete information about known devices and
"nmcli connection show" to get an overview on active connection profiles.

Consult nmcli(1) and nmcli-examples(7) manual pages for complete usage details.

In addition i did install another VM#2 in the KVM with IP 192.168.119.73.
VM#1 192.168.119.74
VM#2 192.168.119.73
Both of them can ping each other
I believe they are communicating via the bridge-242 (br242)

And i tried to ping the VM#1 (192.168.119.74) from other devices that are outside the AlmaLinux. An interesting result was discovered.

PC#A: 192.168.119.11 (able to ping both VM#1 and VM#2)
PC#B: 192.168.119.9 (fail to ping both VM#1 and VM#2)

In tcpdump (both ens224 and br242), i can see the ICMP packet REQUEST from both source address PC#A and PC#B
but VM#1 and VM#2 only REPLY to PC#A, but not PC#B

  • (Success case) If ping from VM#1 to PC#A, tcpdump (both ens224 and br242), shows the ICMP request and reply packet
  • (Failed case) If ping from VM#1 to PC#B, tcpdump (Both ens224 and br242), no ICMP packet was found from VM#1 to PC#B.

In addition, i don’t see any record by using the virsh net-list --all and virsh list --all

So, i am thinking is there any other configure/control could draw this outcome.
Seems like the packet is already arrived ens224 and br242, but VM#1 and VM#2 are replied selectively

6

This seems wrong. You don’t want all traffic from ens224 – from physical wire – to enter the bridge br242.

Next step would be to listen on VM to see whether they receive the ping and how the pong shows (or not).

Yes both are connected to the same bridge, just like physical machines are connected to network switch.

Ideally there would be a DHCP server on the VLAN 242 subnet that would hand out network config for every member of the VLAN (and these VM are members).

Note how the host is member of subnet 10.10.10.0/24. That is also show (and computed from) the IP address notation 10.10.10.74/24

You do have all your 192.168.119.x in 192.168.119.0/24 or something?

The “VMware VMXNET3” … your host is a VM too, so the VM#1 and VM#2 are virtual machines on virtual machine?
While possible, could they not be directly on VMware?

Note: That is from:

[root@host ~]# virsh net-list --all
 Name      State      Autostart   Persistent
----------------------------------------------
 default   active     yes         yes

If you don’t need that subnet, then you can disable the autostart. “Less is more.”

Did you run the command as root? Each user sees their own set of stuff from libvirt.

1 Like
7

@jlehtone thanks for the feedback, yes, you’re right, i need to use the proper user to see the virsh list etc…

++++++
The network problem is getting worse now, i cannot ping both PC#A and PC#B from VM#1 and VM#2
I have done the following capture

tcpdump the bridge(br242) and Ethernet (ens224), then perform a ping test from VM#1 (192.168.119.74) to PC#2 (192.168.119.11)

  • br242: does not see any ICMP packet, but seeing the VM#1 is asking for ARP about PC#2, but not receive any reply
  • ens224: does not see any ICMP packet and ARP packet

PC#2 does not receive any ARP request, in addition the default gateway (192.168.119.9) does not see any ARP from VM#1

I am thinking is there any way to verify the VM#1 has sent out the ARP / ICMP packet via Linux local platform…

8 
tcpdump -nn -v -i ens224 -e

Shows extra data, including VLAN tag of each packet.
One can filter that down to packets of one VLAN. E.g.

tcpdump -nn -v -i ens224 -e vlan 242

One can listen on both vlan and bridge interfaces:

tcpdump -nn -v -i vlan-242 -e
tcpdump -nn -v -i br242 -e

Neither of these should have any vlan tags on packets.

When VM#1 pings PC#2, they are on same subnet, and arp has been forgotten,
VM#1 sends ARP who-has request.
It should show first on br242, then on vlan-242, then on ens224 with vlan tag 242.
If PC#2 replies, then the ARP reply should show
first on ens224 with vlan tag 242, then on vlan-242, then on br242.

You wrote that you see the request on br242, but not on ens224. Does it reach vlan-242?

9

@jlehtone Thanks for the feedback,
I found the the vlan-242 was disappeared, it could be reason why packet won’t be seen in ens224. Therefore, i did re-run the host-net.yml
Then proceed the capture again

+++++
I did clear arp table in VM#1 and then ping test to PC#1 (192.168.119.11) and PC#2 (192.168.119.12)

tcpdump at ens224, vlan-242 and br242
it shows VM#1 sent ARP out, and received MAC addr of both PC#1 and PC#2

Local VM#1 wireshark capture
it shows VM#1 send ARP out, but received ARP of PC#1’s MAC addr only, not received any ARP return about PC#2

Result:
PC#1 ping successfully
PC#2 ping failed.

I have verified the VM#1 (win2022), the firewall is turned off already

In addition, since the ARP is a Broadcast packet, if i wait for few minutes, VM#1 can learn the new IP address of other PC (e.g. PC#3 192.168.119.206) in the same subnet 192.168.119.0/24.

I tried e1000e, hypervisor default “rtl8139” and virtio (driver installed), all of them have the same result. (virtio seems to have a faster response time on PING test, while compare to other 2 device models… )

I have no idea why all the ens224, vlan-242 and br242 can see the ARP reply , but VM#1 cannot receive this missing arp (of PC#2)

10

If most ARP replies do reach VM#1, except that of PC#2, then question is what is different in the reply of PC#2?

11

@jlehtone after the weekend, i found that all VMs and PCs can ping each other.
no extra config or change have been done …
it could be the routing issue, and not the configuration issue.
But no idea where it got stuck/delay.

Thank you so much @jlehtone , finally I understood how does it work from physical switch port to virtual machine’s NIC port, knew how to configure and how to troubleshoot on that.

Thank you again :slight_smile:

1 Like