As the title says, I’m struggling to solve a spontaneous new problem.
I’ve been running this Dell Poweredge T640 Fedora 36 server for ~7 months. I created a br0 bridge in cockpit webUI. I added 2 physical nics to the bridge with no addresses and add manually created libvirt VMs as well as Kubevirt VMs via multus to this bridge regularly. Traffic has been flowing across this bridge no issue the whole time, without issues across updates and reboots until today.
Today I installed updates for the first time in a month, rebooted the server, and now I’m in a bind.
The default (and only route) for the host is via the br0 ip. I can ssh to the server at it’s default route address on the br0 interface. From on the server, I can also ssh to VMs hosted on the server too.
The VMs cannot ping any IP on the LAN other than only the host IP. They cannot ping the default gateway, or anything and I can not reach them from any host on the LAN other than the server they are running on. Also, the two physical pics bridged by br0 are no-longer passing traffic. There are no vlans in play, only one subnet (192.168.1.0/24) this is a really rudimentary network setup. Firewalld is completely disabled. sysctl net forwarding is enabled.
[kc2admin@node1 ~]$ echo; ip a s br0; echo; ip a s enp137s0; echo; ip a s enp132s0f1; echo
10: br0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP group default qlen 1000
link/ether 32:98:e6:96:c7:2d brd ff:ff:ff:ff:ff:ff
inet 192.168.1.21/24 brd 192.168.1.255 scope global noprefixroute br0
valid_lft forever preferred_lft forever
inet6 fe80::d636:cc13:82d0:645a/64 scope link noprefixroute
valid_lft forever preferred_lft forever
4: enp137s0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc mq master br0 state UP group default qlen 1000
link/ether fc:34:97:1e:79:86 brd ff:ff:ff:ff:ff:ff
8: enp132s0f1: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc fq_codel master br0 state UP group default qlen 1000
link/ether a0:36:9f:00:4e:06 brd ff:ff:ff:ff:ff:ff
I could use some troubleshooting ideas. Thank you in advance
I think libvirt et al. might actually use some “lower-level” iptables/nftables commands to configure the routing/firewall rules. You might want to double-check that everything is truly disabled by running iptables --list-rules and/or nft list ruleset.
Yeah. There are some non-trivial forwarding rules being created by the tools there. Unfortunately, I’m not familiar enough with that sort of setup to be of much help. A few basic things to check though would be:
Use brctl show to make sure that the right network interfaces are actually on the right bridges.
Since you said this is using libvirt, make sure that there is a allow br0 line (or whatever your bridge name is) in /etc/qemu/bridge.conf.
If none of that works, I’d probably be down to trial-and-error experimenting with the forwarding rules. I don’t really use libvirt for anything anymore on my systems. I’ve mostly switched to using systemd-nspawn containers and using MACVLAN “bridges” to connect them to the external network. I also use systemd-networkd to configure everything manually so that I know exactly how things are wired and how they are supposed to work. I don’t like it when the tooling auto-configures a lot of stuff that I don’t know about and don’t know how to fix when things go wrong.
Since I started with fedora, I observed all the changes about networking (F28) and it brings my brain cells to glow with all this changes.
If you have time and pleasure I would really appreciate if you could point out (in the discussion mentioned above) how it is on F36 and what exactly you have to turn off to make everything manually as you say you love it.
I’m not against new technologies, not at all. Just love to control them, and not that they control my workflow and interfere in that what i like to configure manually.
I guess I could. It wouldn’t happen anytime soon though since I work as a sysadmin for a state university and the fall semester is approaching rapidly. I have to upgrade a bunch of servers and install a few CMS’s over the next few weeks. But once things settle down, I might.