SELinux AVCs and no network on start: Failed to start systemd-resolved.service

Diagnosis:

journalctl -b -u systemd-resolved.service --no-hostname --no-pager

# systemctl status systemd-resolved.service
Ɨ systemd-resolved.service - Network Name Resolution
     Loaded: loaded (/usr/lib/systemd/system/systemd-resolved.service; enabled; preset: enabled)
     Active: failed (Result: exit-code) since Wed 2023-01-04 06:58:15 CST; 7min ago
       Docs: man:systemd-resolved.service(8)
             man:org.freedesktop.resolve1(5)
             https://www.freedesktop.org/wiki/Software/systemd/writing-network-configuration-managers
             https://www.freedesktop.org/wiki/Software/systemd/writing-resolver-clients
    Process: 1296 ExecStart=/usr/lib/systemd/systemd-resolved (code=exited, status=1/FAILURE)
   Main PID: 1296 (code=exited, status=1/FAILURE)
      Error: 13 (Permission denied)
        CPU: 63ms

# ls -lhat /run/systemd/resolve/
total 0
drwxr-xr-x. 25 root            root            620 Jan  4 06:58 ..
drwxr-xr-x.  2 systemd-resolve systemd-resolve  40 Jan  4 06:58 .

$ journalctl -b -g AVC --no-hostname --no-pager
Jan 04 06:58:15 audit[1258]: AVC avc:  denied  { watch } for  pid=1258 comm="systemd-resolve" path="/" dev="dm-0" ino=128 scontext=system_u:system_r:systemd_resolved_t:s0 tcontext=system_u:object_r:usr_t:s0 tclass=dir permissive=0
Jan 04 06:58:15 audit[1287]: AVC avc:  denied  { watch } for  pid=1287 comm="systemd-resolve" path="/" dev="dm-0" ino=128 scontext=system_u:system_r:systemd_resolved_t:s0 tcontext=system_u:object_r:usr_t:s0 tclass=dir permissive=0
Jan 04 06:58:15 audit[1290]: AVC avc:  denied  { watch } for  pid=1290 comm="systemd-resolve" path="/" dev="dm-0" ino=128 scontext=system_u:system_r:systemd_resolved_t:s0 tcontext=system_u:object_r:usr_t:s0 tclass=dir permissive=0
Jan 04 06:58:15 audit[1293]: AVC avc:  denied  { watch } for  pid=1293 comm="systemd-resolve" path="/" dev="dm-0" ino=128 scontext=system_u:system_r:systemd_resolved_t:s0 tcontext=system_u:object_r:usr_t:s0 tclass=dir permissive=0
Jan 04 06:58:15 audit[1296]: AVC avc:  denied  { watch } for  pid=1296 comm="systemd-resolve" path="/" dev="dm-0" ino=128 scontext=system_u:system_r:systemd_resolved_t:s0 tcontext=system_u:object_r:usr_t:s0 tclass=dir permissive=0

Fix:

# systemctl stop systemd-resolved.service
# systemctl start systemd-resolved.service
# ln -sf /run/systemd/resolve/stub-resolv.conf /etc/resolv.conf

DNS now works as expected, and the files exist under the /run directory as expected:

# ls -lhat /run/systemd/resolve/
total 8.0K
drwxr-xr-x.  3 systemd-resolve systemd-resolve 120 Jan  4 07:09 .
drwx------.  2 systemd-resolve systemd-resolve  80 Jan  4 07:09 netif
-rw-r--r--.  1 systemd-resolve systemd-resolve 912 Jan  4 07:09 resolv.conf
srw-rw-rw-.  1 systemd-resolve systemd-resolve   0 Jan  4 07:09 io.systemd.Resolve
-rw-r--r--.  1 systemd-resolve systemd-resolve 920 Jan  4 07:09 stub-resolv.conf
drwxr-xr-x. 26 root            root            640 Jan  4 07:09 ..

Iā€™m unsure why there would be SELinux errors during boot but not when I manually restart the service.

Selinux errors can often be fixed by running either sudo touch /.autorelabel followed by a reboot, or sudo restorecon -R / which checks and adjusts the context while running. There is also the fixfiles command.

I have had a couple instances where selinux policies have been updated and caused similar problems because while the policy was updated and in effect after a boot, the actual file contexts were not updated automatically, thus causing errors.

I have often thought that the context updates should be forced when selinux policies are changed.

2 Likes

Thanks @computersavvy
Caught this one by adding v to show changes:

restorecon -Rv /
...
Relabeled /run/NetworkManager/no-stub-resolv.conf from system_u:object_r:NetworkManager_var_run_t:s0 to system_u:object_r:net_conf_t:s0
2 Likes

They used to be and it would take forever on your next reboot back when everyone had a spinning disk.

Yes, that is kind of a catch-22 situation.
Delay on boot if the context update is forced ā€“ vs ā€“ some apps that may not work if the delay is not forced. Personally I think there may be less confusion by forcing the context relabel.

At least the user should be told that the policy has been updated and a relabel suggested whenever selinux policies are updated. That would allow the user the choice of when to relabel if they were aware of the need.

Might be an interesting use case for adding it to the needs-restarting tool and perhaps adding a var for it in dnf.conf to touch /.autorelabel if set.