Problem setting up a router with PPPoE using NetworkManager and firewalld

[nibsniven]

I’ve been struggling to build a simple router with PPPoE and a basic firewall. I need to use Fedora 33 server with no custom software.

I’m using Fedora 33 with NetworkManager (nmcli) and firewalld (firewall-cmd) on an i5 box with 8GB ram. I have PPPoE to a bridged DSL modem which connects fine. I have an external IP on eno1->ppp0. I have a LAN NIC (enps1f0s0) with IP 10.0.0.1 that acts as a gateway for 4 other hosts, all of which can get to the internet. The problem is nothing from the outside sees the router. I can’t ping it from the internet, nor access port 80 which is open on the router. No port forwarding works. I can see the traffic getting to eno1 via tcpdump (ICMP and http) but nothing gets back out from the router, or forwarded to LAN hosts.

Is there anyone who can help out? I’ve been scouring the limited documentation on this issue for days now.

[mattdm (Matthew Miller)]

Just for clarity, do outgoing connections from the router system work? You’re just not getting incoming traffic?

[nibsniven]

Thanks for the reply. Yes, all is well from connections initiated from the router machine, and also all the machines using it as a gateway. Only new connections from the internet to the router are a problem, although they appear in a tcpdump of the NIC, so the packets are getting there, just not picked up by the services, or forwarded through NAT.

An nmap of the machine from the internet shows no ping replies, nor any open ports, but ICMP echo is turned on, as is nginx on port 80. Pings from the LAN work, as does port 80.

As an aside, I followed what little I could find on the web about dong this, but most of it used iptables, and/or rp-ppoe. None used nft with firewalld, and NetworkManager-ppp. Frustrating. I can’t be the only one who wants to do this using the latest tools. Frustrating!

[computersavvy (JV)]

The gateway router must have port forwarding enabled for outside entities to contact those inside, and then they use the outward facing IP:port method and that port can be forwarded to only 1 internal host… Since you are on a private LAN (10.0.0.X is a private address) the internet cannot connect to your hosts directly.

The way NAT works is that it allows answers to outgoing established connections to be returned, but blocks new connections from outside. If everything from the inside is working well with the internet I really do not understand the problem.

Oh, BTW, the router host must have NAT and forwarding enabled and on mine it is automatically enabled.

[sideburns (Joe Zeff)]

What IP are you trying to ping from outside your LAN? If you’re trying to ping one of your inside machines that’s just not going to word, because they’re all using non-routable IP addresses, and if you’re trying to ping your router’s public IP, that will either work or fail depending on how your router’s configured. It sounds to me like everything’s working exactly like it’s supposed to.

[mattdm (Matthew Miller)]

Ok, so… what does your firewall configuration look like? It sounds like you have forwarding (and possibly NAT) already set up?

[ilikelinux]

@nibsniven

An nmap of the machine from the internet shows no ping replies, nor any open ports, but ICMP echo is turned on, as is nginx on port 80. Pings from the LAN work, as does port 80.

I guess you have to do a port forward from the incoming traffic WAN>LAN for every service.
As example your external ip 88.8.8.8:22 to internal 10.0.0.2:2222 and so on.

If this not works you have to check with your ISP if he not blocks this services for incoming traffic.

[vgaetera (Vladislav Grigoryev)]

I use systemd-networkd and firewalld on a Fedora 33 VPS that works as a router.
It’s a bit tricky because policy objects required for zone-to-zone forwarding are available only in the next code branch starting with firewalld-0.9.0 which is supposed to be only in Fedora 34.
So, we need to deal with both rich and direct rules assuming that the default backend is nftables:

WAN_ZONE="external"
WAN_CON="teksavvy"
WAN_IF="teklink"
WAN_IF2="ppp0"
LAN_ZONE="internal"
LAN_CON="tekbridge"
LAN_IF="nm-bridge"
sudo nmcli connection modify id ${WAN_CON} connection.zone ${WAN_ZONE}
sudo nmcli connection down id ${WAN_CON}
sudo nmcli connection up id ${WAN_CON}
sudo nmcli connection modify id ${LAN_CON} connection.zone ${LAN_ZONE}
sudo nmcli connection down id ${LAN_CON}
sudo nmcli connection up id ${LAN_CON}
sudo firewall-cmd --permanent --zone=${WAN_ZONE} --set-target=ACCEPT
sudo firewall-cmd --permanent --zone=${WAN_ZONE} --add-masquerade
sudo firewall-cmd --permanent --zone=${WAN_ZONE} --add-rich-rule="rule protocol value=icmp accept"
sudo firewall-cmd --permanent --zone=${WAN_ZONE} --add-rich-rule="rule protocol value=ipv6-icmp accept"
sudo firewall-cmd --permanent --zone=${WAN_ZONE} --add-rich-rule="rule priority=32767 reject"
sudo firewall-cmd --permanent --zone=${LAN_ZONE} --set-target=ACCEPT
sudo firewall-cmd --permanent --zone=${LAN_ZONE} --add-rich-rule="rule protocol value=icmp accept"
sudo firewall-cmd --permanent --zone=${LAN_ZONE} --add-rich-rule="rule protocol value=ipv6-icmp accept"
sudo firewall-cmd --permanent --zone=${LAN_ZONE} --add-rich-rule="rule priority=32767 reject"
sudo firewall-cmd --permanent --direct --remove-rules ipv4 mangle FORWARD
sudo firewall-cmd --permanent --direct --add-rule ipv4 mangle FORWARD 10 -p tcp --syn -j TCPMSS --clamp-mss-to-pmtu
sudo firewall-cmd --permanent --direct --remove-rules ipv4 filter FORWARD
sudo firewall-cmd --permanent --direct --add-rule ipv4 filter FORWARD 10 -i lo -j ACCEPT
sudo firewall-cmd --permanent --direct --add-rule ipv4 filter FORWARD 20 -i ${LAN_IF} -j ACCEPT
sudo firewall-cmd --permanent --direct --add-rule ipv4 filter FORWARD 30 -i ${WAN_IF} -m conntrack --ctstate RELATED,ESTABLISHED,DNAT -j ACCEPT
sudo firewall-cmd --permanent --direct --add-rule ipv4 filter FORWARD 30 -i ${WAN_IF2} -m conntrack --ctstate RELATED,ESTABLISHED,DNAT -j ACCEPT
sudo firewall-cmd --permanent --direct --add-rule ipv4 filter FORWARD 40 -i ${WAN_IF} -p icmp -m conntrack --ctstate NEW -j ACCEPT
sudo firewall-cmd --permanent --direct --add-rule ipv4 filter FORWARD 40 -i ${WAN_IF2} -p icmp -m conntrack --ctstate NEW -j ACCEPT
sudo firewall-cmd --permanent --direct --add-rule ipv4 filter FORWARD 80 -i ${WAN_IF} -m conntrack --ctstate INVALID -j DROP
sudo firewall-cmd --permanent --direct --add-rule ipv4 filter FORWARD 80 -i ${WAN_IF2} -m conntrack --ctstate INVALID -j DROP
sudo firewall-cmd --permanent --direct --add-rule ipv4 filter FORWARD 90 -j REJECT --reject-with icmp-admin-prohibited
sudo firewall-cmd --permanent --direct --remove-rules ipv6 mangle FORWARD
sudo firewall-cmd --permanent --direct --add-rule ipv6 mangle FORWARD 10 -p tcp --syn -j TCPMSS --clamp-mss-to-pmtu
sudo firewall-cmd --permanent --direct --remove-rules ipv6 filter FORWARD
sudo firewall-cmd --permanent --direct --add-rule ipv6 filter FORWARD 10 -i lo -j ACCEPT
sudo firewall-cmd --permanent --direct --add-rule ipv6 filter FORWARD 20 -i ${LAN_IF} -j ACCEPT
sudo firewall-cmd --permanent --direct --add-rule ipv6 filter FORWARD 30 -i ${WAN_IF} -m conntrack --ctstate RELATED,ESTABLISHED,DNAT -j ACCEPT
sudo firewall-cmd --permanent --direct --add-rule ipv6 filter FORWARD 30 -i ${WAN_IF2} -m conntrack --ctstate RELATED,ESTABLISHED,DNAT -j ACCEPT
sudo firewall-cmd --permanent --direct --add-rule ipv6 filter FORWARD 40 -i ${WAN_IF} -p ipv6-icmp -m conntrack --ctstate NEW -j ACCEPT
sudo firewall-cmd --permanent --direct --add-rule ipv6 filter FORWARD 40 -i ${WAN_IF2} -p ipv6-icmp -m conntrack --ctstate NEW -j ACCEPT
sudo firewall-cmd --permanent --direct --add-rule ipv6 filter FORWARD 80 -i ${WAN_IF} -m conntrack --ctstate INVALID -j DROP
sudo firewall-cmd --permanent --direct --add-rule ipv6 filter FORWARD 80 -i ${WAN_IF2} -m conntrack --ctstate INVALID -j DROP
sudo firewall-cmd --permanent --direct --add-rule ipv6 filter FORWARD 90 -j REJECT --reject-with icmp6-adm-prohibited
sudo firewall-cmd --reload

[nibsniven]

$ for z in $(firewall-cmd --get-zones);do sudo firewall-cmd --zone=${z} --list-all;done
FedoraServer
  target: default
  icmp-block-inversion: no
  interfaces: 
  sources: 
  services: cockpit dhcpv6-client ssh
  ports: 
  protocols: 
  masquerade: no
  forward-ports: 
  source-ports: 
  icmp-blocks: 
  rich rules: 
FedoraWorkstation
  target: default
  icmp-block-inversion: no
  interfaces: 
  sources: 
  services: dhcpv6-client samba-client ssh
  ports: 1025-65535/udp 1025-65535/tcp
  protocols: 
  masquerade: no
  forward-ports: 
  source-ports: 
  icmp-blocks: 
  rich rules: 
block
  target: %%REJECT%%
  icmp-block-inversion: no
  interfaces: 
  sources: 
  services: 
  ports: 
  protocols: 
  masquerade: no
  forward-ports: 
  source-ports: 
  icmp-blocks: 
  rich rules: 
dmz
  target: default
  icmp-block-inversion: no
  interfaces: 
  sources: 
  services: ssh
  ports: 
  protocols: 
  masquerade: no
  forward-ports: 
  source-ports: 
  icmp-blocks: 
  rich rules: 
drop
  target: DROP
  icmp-block-inversion: no
  interfaces: 
  sources: 
  services: 
  ports: 
  protocols: 
  masquerade: no
  forward-ports: 
  source-ports: 
  icmp-blocks: 
  rich rules: 
external
  target: default
  icmp-block-inversion: no
  interfaces: 
  sources: 
  services: http
  ports: 514/tcp
  protocols: 
  masquerade: yes
  forward-ports: 
	port=514:proto=tcp:toport=514:toaddr=10.0.0.5
  source-ports: 
  icmp-blocks: echo-reply
  rich rules: 
home (active)
  target: ACCEPT
  icmp-block-inversion: no
  interfaces: nm-bridge
  sources: 
  services: dhcpv6-client mdns samba-client ssh
  ports: 
  protocols: 
  masquerade: no
  forward-ports: 
  source-ports: 
  icmp-blocks: 
  rich rules: 
	rule protocol value="icmp" accept
	rule protocol value="ipv6-icmp" accept
	rule priority="32767" reject
internal (active)
  target: default
  icmp-block-inversion: no
  interfaces: enp1s0f0 enp1s0f1 enp2s0f0 enp2s0f1
  sources: 
  services: http ssh
  ports: 
  protocols: 
  masquerade: no
  forward-ports: 
  source-ports: 
  icmp-blocks: 
  rich rules: 
	rule family="ipv4" source address="10.0.0.0/24" masquerade
nm-shared
  target: ACCEPT
  icmp-block-inversion: no
  interfaces: 
  sources: 
  services: dhcp dns ssh
  ports: 
  protocols: icmp ipv6-icmp
  masquerade: no
  forward-ports: 
  source-ports: 
  icmp-blocks: 
  rich rules: 
	rule priority="32767" reject
public (active)
  target: ACCEPT
  icmp-block-inversion: no
  interfaces: eno1
  sources: 
  services: dhcpv6-client http mdns ssh
  ports: 
  protocols: 
  masquerade: yes
  forward-ports: 
  source-ports: 
  icmp-blocks: echo-reply
  rich rules: 
	rule protocol value="icmp" accept
	rule protocol value="ipv6-icmp" accept
	rule priority="32767" reject
trusted
  target: ACCEPT
  icmp-block-inversion: no
  interfaces: 
  sources: 
  services: 
  ports: 
  protocols: 
  masquerade: no
  forward-ports: 
  source-ports: 
  icmp-blocks: 
  rich rules: 
work
  target: default
  icmp-block-inversion: no
  interfaces: 
  sources: 
  services: dhcpv6-client mdns ssh
  ports: 
  protocols: 
  masquerade: no
  forward-ports: 
  source-ports: 
  icmp-blocks: 
  rich rules:

[nibsniven]

nmcli

enp1s0f1: connected to enp1s0f1
	"Intel 82571EB/82571GB D0/D1"
	ethernet (e1000e), 00:1F:29:60:95:4B, hw, mtu 1500
	ip4 default, ip6 default
	inet4 192.168.1.2/24
	route4 192.168.1.0/24
	route4 0.0.0.0/0
	inet6 fe80::21f:29ff:fe60:954b/64
	route6 fe80::/64
	route6 2001:1970:5c1d:b900::/64
	route6 ::/0
	route6 ff00::/8

nm-bridge: connected to bridge
	"nm-bridge"
	bridge, 46:61:26:73:E0:D6, sw, mtu 1500
	inet4 10.0.0.1/24
	route4 10.0.0.0/24
	route4 76.10.128.233/32
	route4 0.0.0.0/0
	inet6 fe80::2d9f:591b:a207:a851/64
	route6 fe80::/64
	route6 ff00::/8

eno1: connected to teksavvy
	"Intel 82579LM"
	ethernet (e1000e), 3C:D9:2B:5C:8E:8D, hw, iface ppp0, mtu 1500
	inet4 xxx.xxx.xxx.xxx/32
	route4 0.0.0.0/0
	route4 xxx.xxx.xxx.xxx/32
	inet6 xxx/10
	route6 ff00::/8
	route6 fe80::/10
	route6 ::/0
	route6 fe80::/10

enp1s0f0: connected to br-slave-1
	"Intel 82571EB/82571GB D0/D1"
	ethernet (e1000e), 00:1F:29:60:95:4A, hw, mtu 1500
	master nm-bridge

enp2s0f0: connected to br-slave-2
	"Intel 82571EB/82571GB D0/D1"
	ethernet (e1000e), 00:15:17:6B:05:CA, hw, mtu 1500
	master nm-bridge

enp2s0f1: connected to br-slave-3
	"Intel 82571EB/82571GB D0/D1"
	ethernet (e1000e), 00:15:17:6B:05:CB, hw, mtu 1500
	master nm-bridge

ppp0: disconnected
	"ppp0"
	ppp, sw, mtu 1492

lo: unmanaged
	"lo"
	loopback (unknown), 00:00:00:00:00:00, sw, mtu 65536

[nibsniven]

Thanks for your reply. Unfortunately your script failed to solve my problem. Everything from outside is still ignored.

[nibsniven]

Thanks for your reply. My ISP isn’t blocking anything; outside packets show up on eno1 according to tcpdump.

[nibsniven]

@sideburns

What IP are you trying to ping from outside your LAN? If you’re trying to ping one of your inside machines that’s just not going to word, because they’re all using non-routable IP addresses, and if you’re trying to ping your router’s public IP, that will either work or fail depending on how your router’s configured. It sounds to me like everything’s working exactly like it’s supposed to.

Thanks for your reply. I’m pinging the IP given by my ISP. It’s not working as it’s supposed to because I want to be able to get a reply to pings, and also access port 80 from outside. The packets get to the NIC, but are all ignored.

[nibsniven]

@computersavvy

The gateway router must have port forwarding enabled for outside entities to contact those inside, and then they use the outward facing IP:port method and that port can be forwarded to only 1 internal host… Since you are on a private LAN (10.0.0.X is a private address) the internet cannot connect to your hosts directly.

The way NAT works is that it allows answers to outgoing established connections to be returned, but blocks new connections from outside. If everything from the inside is working well with the internet I really do not understand the problem.

Oh, BTW, the router host must have NAT and forwarding enabled and on mine it is automatically enabled.

Thanks for your reply. My machine is the gateway router host, and has a forwarded port as well as ping enabled. On my machine it isn’t automatically enabled, or at least it isn’t working as it should.

[nibsniven]

@vgaetera

Do not assign bridge-slave interfaces to any zones. You should assign only the master connection.

I agree, but every time I unassign those NICs, NetworkManager reassigns it again.

The interface is under control of NetworkManager, setting zone to default.

How can I get around that?

@vgaetera

Move this port forwarfing to the public zone since you have configured it as the WAN zone.

Done.

@vgaetera

In addition, since you are using NetworkManager, you don’t need to assign interfaces in the firewalld config, instead modify the connections.

Done. Thanks for your help. Unfortunately I still can’t ping or reach port 80 from outside. Port forwarding to 10.0.0.5 via 514 is blocked.

[vgaetera (Vladislav Grigoryev)]

Let’s check your configs:

sudo grep -r -e interface /etc/firewalld/zones; \
sudo grep -r -e zone /etc/NetworkManager/system-connections

nmcli -g name connection show | while read NM_CON; \
do nmcli -f connection.id,connection.zone connection show id ${NM_CON}; done

sudo firewall-cmd --get-active-zone | sed -e "/^\s/d" \
| while read -r FW_ZONE; do sudo firewall-cmd --zone=${FW_ZONE} --list-all; done

sudo firewall-cmd --direct --get-all-chains; \
sudo firewall-cmd --direct --get-all-rules; \
sudo firewall-cmd --direct --get-all-passthroughs

[nibsniven]

@vgaetera

grep -r -e interface /etc/firewalld/zones

/etc/firewalld/zones/internal.xml:  <interface name="nm-bridge"/>

@vgaetera

grep -r -e zone /etc/NetworkManager/system-connections

/etc/NetworkManager/system-connections/enp2s0f0.nmconnection:zone=internal
/etc/NetworkManager/system-connections/enp1s0f1.nmconnection:zone=internal
/etc/NetworkManager/system-connections/enp2s0f1.nmconnection:zone=internal
/etc/NetworkManager/system-connections/enp1s0f0.nmconnection:zone=internal
/etc/NetworkManager/system-connections/eno1.nmconnection:zone=external
/etc/NetworkManager/system-connections/bridge.nmconnection:zone=home
/etc/NetworkManager/system-connections/br-slave-3-4faa3c37-b9dd-48c6-bc6e-658357c6f945.nmconnection:zone=internal
/etc/NetworkManager/system-connections/br-slave-2-d0ea0f74-e014-46e5-987f-0e8b60f35517.nmconnection:zone=internal
/etc/NetworkManager/system-connections/teksavvy.nmconnection:zone=public

@vgaetera

nmcli -g name connection show | while read NM_CON; \
do nmcli -f connection.id,connection.zone connection show id ${NM_CON}; done

connection.id:                          enp1s0f1
connection.zone:                        internal
connection.id:                          bridge
connection.zone:                        home
connection.id:                          teksavvy
connection.zone:                        public
connection.id:                          br-slave-1
connection.zone:                        --
connection.id:                          br-slave-2
connection.zone:                        --

connection.id:                          br-slave-2
connection.zone:                        internal
connection.id:                          br-slave-3
connection.zone:                        --

connection.id:                          br-slave-3
connection.zone:                        internal
connection.id:                          br-slave-2
connection.zone:                        --

connection.id:                          br-slave-2
connection.zone:                        internal
connection.id:                          br-slave-3
connection.zone:                        --

connection.id:                          br-slave-3
connection.zone:                        internal
connection.id:                          eno1
connection.zone:                        external
connection.id:                          enp1s0f0
connection.zone:                        internal
connection.id:                          enp2s0f0
connection.zone:                        internal
connection.id:                          enp2s0f1
connection.zone:                        internal

@vgaetera

firewall-cmd --get-active-zone | sed -e "/^\s/d" | \
while read -r FW_ZONE; do sudo firewall-cmd --zone=${FW_ZONE} --list-all; done

home (active)
  target: ACCEPT
  icmp-block-inversion: no
  interfaces: nm-bridge
  sources: 
  services: dhcpv6-client mdns samba-client ssh
  ports: 
  protocols: 
  masquerade: no
  forward-ports: 
  source-ports: 
  icmp-blocks: 
  rich rules: 
	rule protocol value="icmp" accept
	rule protocol value="ipv6-icmp" accept
	rule priority="32767" reject
internal (active)
  target: default
  icmp-block-inversion: no
  interfaces: enp1s0f0 enp1s0f1 enp2s0f0 enp2s0f1
  sources: 
  services: http ssh
  ports: 
  protocols: 
  masquerade: no
  forward-ports: 
  source-ports: 
  icmp-blocks: 
  rich rules: 
	rule family="ipv4" source address="10.0.0.0/24" masquerade
public (active)
  target: ACCEPT
  icmp-block-inversion: no
  interfaces: eno1
  sources: 
  services: http
  ports: 
  protocols: 
  masquerade: yes
  forward-ports: 
	port=514:proto=udp:toport=:toaddr=10.0.0.5
  source-ports: 
  icmp-blocks: echo-reply
  rich rules: 
	rule protocol value="icmp" accept
	rule protocol value="ipv6-icmp" accept
	rule priority="32767" reject

@vgaetera

firewall-cmd --direct --get-all-chains

Nothing returned.

@vgaetera

firewall-cmd --direct --get-all-rules

ipv4 filter FORWARD 0 -i lo -j ACCEPT
ipv4 filter FORWARD 0 -i nm-bridge -j ACCEPT
ipv4 filter FORWARD 0 -i eno1 -m conntrack --ctstate RELATED,ESTABLISHED,DNAT -j ACCEPT
ipv4 filter FORWARD 0 -i eno1 -p icmp -m conntrack --ctstate NEW -j ACCEPT
ipv4 filter FORWARD 0 -i eno1 -m conntrack --ctstate INVALID -j DROP
ipv4 filter FORWARD 0 -j REJECT --reject-with icmp-admin-prohibited
ipv6 filter FORWARD 0 -i lo -j ACCEPT
ipv6 filter FORWARD 0 -i nm-bridge -j ACCEPT
ipv6 filter FORWARD 0 -i eno1 -m conntrack --ctstate RELATED,ESTABLISHED,DNAT -j ACCEPT
ipv6 filter FORWARD 0 -i eno1 -p ipv6-icmp -m conntrack --ctstate NEW -j ACCEPT
ipv6 filter FORWARD 0 -i eno1 -p tcp -m tcp --dport 51413 -m conntrack --ctstate NEW -j ACCEPT
ipv6 filter FORWARD 0 -i eno1 -p udp -m udp --dport 51413 -m conntrack --ctstate NEW -j ACCEPT
ipv6 filter FORWARD 0 -i eno1 -m conntrack --ctstate INVALID -j DROP
ipv6 filter FORWARD 0 -j REJECT --reject-with icmp6-adm-prohibited

@vgaetera

firewall-cmd --direct --get-all-passthroughs

ipv4 -I FORWARD -p tcp --tcp-flags SYN,RST SYN -j TCPMSS --clamp-mss-to-pmtu

Thanks for your help!

[vgaetera (Vladislav Grigoryev)]

It looks like there’s several duplicated connections.
It’s best to remove inactive doubles by UUID to avoid conflicts with bridge interface members.
Check all connections:

nmcli connection show

[sideburns (Joe Zeff)]

If memory serves, pings are handled by the router itself, rather than being passed on to any internal machine. Is your router set to respond or to drop them? You may need to ask your ISP about this.

[mattdm (Matthew Miller)]

I think in this case there is no separate router/gateway device. This Linux box is being set up to be exactly that. A lot of devices these days are all-in-one modem, router/nat gateway, and wifi access point. But as I understand the setup here, it’s a DSL modem connected directly to the system in question, which will be the router when everything is working properly.