Dnf upgrade of some packages fail after upgrade from F35

, ,

Problem

On some systems upgraded from Fedora Linux 35 cannot upgrade certain packages.
Error message given by dnf upgrade is like this:

  Upgrading        : flatpak-1.12.6-1.fc36.x86_64                                                                                                                                         4/8 
error: lsetfilecon: (/usr/libexec/flatpak-system-helper;6232783d, system_u:object_r:flatpak_helper_exec_t:s0) Invalid argument
error: Plugin selinux: hook fsm_file_prepare failed

Error unpacking rpm package flatpak-1.12.6-1.fc36.x86_64

Known affected packages include crun, containers-common, conmon, podman, swtpm, runc, flatpak.

Doing SELinux relabelling does not fix the problem.
Even worse, attempting to recover by doing SELinux relabelling with fixfiles -F onboot
(or touch /.autorelabel, which is the same thing)
may result in a system where login is impossible.
After entering the password, the login screen is shown again.

Cause

Not known yet.

Related Issues

Bugzilla report: #2056303
Bugzilla report: #2069102

Workarounds

Reinstalling packages that provide the problematic SELinux modules,
then doing a SELinux relabel fixes the issue.
Because the modules have interdependencies,
they need to be all removed in a single transaction.

  1. Check what custom SELinux modules are installed:
    semodule -lfull | grep -v 100
  2. Check which packages provide these modules, e.g.
    dnf provides /usr/share/selinux/packages/swtpm.pp.
    The file may also have extension .pp.bz2 and may reside in a subdirectory.
    Write the list of packages, so that you can reinstall them in a later step.
    If there are modules that do not come from any package,
    e.g. because you have used audit2allow to silence earlier SELinux denials,
    you need to resolve each case individually.
    It is possible that the module does not need to be touched at all.
  3. Remove all found custom modules, e.g.
    semodule -X 200 -r container -X 200 -r swtpm
  4. dnf reinstall selinux-policy selinux-policy-targeted
  5. Reinstall packages that provide the custom modules from step 3,
    e.g. dnf reinstall swtpm
  6. Set up full relabel with fixfiles -F onboot and reboot.

To recover from the state where login is not possible,
use some recovery method to set SELinux to permissive,
boot and login, then relabel again with restorecon -vR /.
After that, login should work even after setting SELinux back to enforcing.
The above sequence of reinstalling custom SELinux modules still needs to be performed after this.

2 Likes

I have been getting a lot of repeted avc denials, every few seconds since my last update, even on fedora 34.


Following your workaround using restorecon I have gotten thousands of context relabeling messages, both under my home directory and in several of the common system directories.
I added the f34 tag to this since it showed up on my f34 daily driver with a reboot today.

It seems a major context change was done and relabeling was not automatically done, so that a reboot to reload the selinux policies without relabeling triggered the errors, Including this one.
Relabeled /boot/grub2/grub.cfg from system_u:object_r:dosfs_t:s0 to system_u:object_r:boot_t:s0
That by itself may prevent booting if selinux is enforcing. I have selinux as permissive so have not been blocked from booting.

Well, at the very least this will be an interesting test of my bot’s ability to handle multiple bugzilla links. It should work… we will find out!

1 Like

Thanks @oturpe for creating this report. Let’s wait a few days if the bug reports get some movement or if we can somehow identify when exactly this happens. I couldn’t replicate this, personally.

3 Likes

Could not update flatpak either.

Running transaction
Preparing : 1/1
Running scriptlet: flatpak-1.12.7-1.fc36.x86_64 1/2
Upgrading : flatpak-1.12.7-1.fc36.x86_64 1/2
error: lsetfilecon: (/usr/libexec/flatpak-system-helper;6246dcac, system_u:object_r:flatpak_helper_exec_t:s0) Invalid argument
error: Plugin selinux: hook fsm_file_prepare failed

Error unpacking rpm package flatpak-1.12.7-1.fc36.x86_64
Verifying : flatpak-1.12.7-1.fc36.x86_64 1/2
Verifying : flatpak-1.12.7-1.fc35.x86_64 2/2

1 Like

Please post that info into bugzilla. This is a place to document the issue, not debug it. Thanks.

1 Like

Posted in bugzilla as well.

2 Likes

Just an update.
The relabeling involved flatpak on my two f35 machines as well as my f34 machine. It also involved grub (/boot/grub2 and below) and /var/lib/boinc on all 3 machines, and /boot + /boot/extlinux on one machine.

I am now ready to check my f36 VM to see if it occurs there as well.

Whatever selinux update was done really should have done the relabel for the user so it did not break booting. By relabeling before I reboot it allows the boot to complete successfully.

Posted into bugzilla, a workaround that worked for me was:

sudo semodule -X 200 -r snappy -r container -r flatpak -X 400 -r pcpupstream -r pcpupstream-container -X 100 -r pcp
sudo dnf remove pcp
sudo dnf reinstall -y container-selinux
sudo dnf upgrade

The actual semodule command above may depend on what other packages you have installed, I had to add the pcp ones after semodule told me it couldn’t remove it because it was in use by another module (had to keep adding more modules until it was happy, you can find out the module names and priorities with semodule -lfull).

Note that during the removal of pcp it got stuck (pmcd and auditd using 100% CPU), so I had to stop it with systemctl stop pmcd in order for dnf remove pcp to succeed.

2 Likes

@edwintorok which bugzilla report did you post your workaround in?

Maybe related to Flatpak issue after system-upgrade to F36

My workaround: 2056303 – After upgrade to 36 several packages fail to upgrade to fc36 and 2056303 – After upgrade to 36 several packages fail to upgrade to fc36
Others have posted their workaround on the bug too, the actual list of modules that have to be removed using semodule varies a bit (probably due to what other packages users got installed or not)

1 Like

I’d like to promote this topic into the main #common-issues category, but I’m not proficient with SELinux enough to judge whether the proposed workarounds are good (e.g. safe for users systems, without side effects) or not. I asked the selinux maintainer to share his views, if possible.

I notice at least one flaw. My system has a custom module I made from audit2allow and added myself. This has priority 400, so would be included with grep -v 100, but isn’t provided by a package. I’m not sure if that needs to be re-installed, but it wouldn’t be with dnf reinstall. So, this should either:

  1. use grep ^200 instead oif grep -v 100? This would exclude 400 (leaving it alone)
  2. use the grep -v 100 and remove anything that has some other priority, and include instructions on putting it back.

I found SELinux/IndependentPolicy - Fedora Project Wiki β€” but I’m not sure if that’s authoritative. (It notes that it is not official Fedora packaging policy.) Looks like there are examples in the bug with various priorities like 300 and 400 :person_shrugging:.

(Maybe this should be official policy?)

A package update which may fix the issue in bug 2056303 has been submitted to the Fedora updates system.

Stay tuned β€” it will be available for testing soon!

A candidate fix for this issue has been submitted to the updates-testing repository for testing. If you are experiencing this problem, please help test this update and then report to Bodhi if it solves the problem β€” or not!

To install this update on your system, run this command: sudo dnf --enablerepo=updates-testing update --advisory=FEDORA-2022-c5bee6b70f

I updated the instructions a bit to mention the case of modules not coming from any package.
Of course, such modules can be anything,
so it is impossible to give guidance that always works.
So just say that each case needs to be resolved individually,
but it is possible that they do not have to be touched at all.

The bug has now been marked as a Final blocker,
and there seems to be a fix that does not need user interaction, too.
So perhaps there is no need to have it as a Common Issue anymore?

Your issue seems to be something different than what this entry is supposed to document.
The issue here is that after the upgrade to F36,
dnf upgrade fails for SELinux related reasons,
and just relabelling everything does not fix the issue.

There can be other issues related to SELinux, too.
Not all of the should be grouped together.

If it is a highly visible issue and I have time, I generally include even those which are going to be fixed, because it helps people find the issue description and the fix (even if it means just β€œupdate your system”).

For this one, I’d promote it immediately if it listed no workarounds and just said β€œplease wait until it’s fixed”. But I don’t feel comfortable including workarounds which I don’t fully understand, e.g. which might have some long-lasting negative effects on the system (like removing selected selinux modules). That’s why I asked the developer for looking at it (but he seems to have little time for this, unfortunately).

An update has been released to fix this issue.

A new F35 selinux-policy build. This build should help with F35->F36 upgrade problems.

After you update your system in your usual way (and possibly reboot), you should no longer be affected by this problem. If the problem persists, please start a new discussion topic and we’ll help figure out what’s still wrong.

Please, trying this : Problem update packages in fedora 36 - #2 by ja7ad

1 Like