Initially this started as a normal Fedora 34 upgrade for four systems. First issue noticed was the re-occurring hardware watch dog issue which has occurred over many Fedora versions but usually would only affect one system and then when we tried diagnosing it would go away. Yes bug reports were filed. This time all four systems had the issue which can be worked around by editing /etc/systemd/system.conf file. This started the thinking about the systems being upgraded for 10 or more years and maybe the issue would go away with a fresh install. Well on a reboot one system had a corruption which failed to boot.
Configuration is an Asus H971Plus mother board with two 1G hardware raided SSD drives and a MyPassport USB portable external drive for disaster recovery. Both the internal SSD and external MyPassport drive have Fedora 34 installed. The purpose of the portable drive is to store DD images of the running systems for a quick recovery so it has a live Fedora install with a large partition for storing image files for the four systems.
First it appears that Fedora now knows about the hardware raid configuration and has created /dev/md device entries which causes duplicate path issues in the LVM file system which needed a filter added /etc/lvm/lvm.conf to not scan the /dev/sda device to get rid of the multiple paths for the LVM and then needed to do a vgck --updatemetadata fedora_redwood to fix the volume headers as the preferred device is now the /dev/md device.
The real issue now is after at least 10 installs of Fedora 34 over the two weeks is the system looks fine until a reboot and it ends up back to where to what prompted the new install the boot fails and you end up in rescue mode at the grub prompt to enter the root password. Digging around I found that Fedora 34 has made changes to the UEFI boot which appears to be the issue.
If I boot the system without the portable drive attached the boot fails and logged into the rescue system if the efibootmgr command is run there is no EFI data. Suspect that information was put on the portable drive as a efibootmgr command returns EFI data. Yet it must be wrong as the system will still fail to boot.
So the question is how to get out of this pickle? Is it possible to fix the missing EFI data issue on the internal SSD storage without doing another install? If it can be done without another install where do you get the information to fix it? How to prevent this issue in the future?
Another issue to be further dug into is the mother board setup boot order seems to get changed by discovery of boot able devices being plugged in.