This is the issue.
/boot by default during an automatic install is created as an ext4 partition. Also /boot/efi is also created by default as the ESP (vfat) partition if not already there.
In order for grub to access a btrfs subvolume as /boot it must load the btrfs module from the ESP partition before the kernel is even loaded. The kernel is the vmlinuz file it is complaining about.
I suggest that for simplicity and to eliminate the problems with trying to get the bios to access a btrfs subvolume that you reinstall with the standard partitioning.
/boot/efi as ESP formatted vfat
/boot as ext4
btrfs volume for the rest with subvolumes / and /home as default.
This will be the standard structure if you use automatic partitioning during the install of the workstation version. AFAIK the server version does not use btrfs as of yet.
/boot/efi is of course an ESP FAT32 partition.
/boot as a btrfs subvolume was supported according to documentation (apart from some encryption related issue, if I remember correctly, which did not apply to my use case – it could be something else but I remember clearly that it didn’t apply to my use case when I selected this scheme) and working perfectly fine in F35. I manually chose this scheme to keep everything apart from /boot/efi (which is standard as ESP FAT32) in btrfs. I did not and do not ever use auto partitioning when deploying Fedora or any other distro.
Reinstalling anything is out of the question. This is a production server. I could resize the partition and switch /boot to ext4 manually but the point was to stick with “going forward” with btrfs and drop ext4 entirely which is what Fedora aimed for according to everything I’ve read when the default fs change was announced. Apart from the message in dnf, which seems to be a grubby issue, the kernels boot properly, so I’m not sure there’s anything wrong here apart from a cosmetic issue. My intent with this post was to identify exactly what is wrong so I can make a proper bug report (probably on grubby).
I have seen a few other posts about /boot as btrfs.
IIRC mostly it has been problematic in that before grub can read anything from the btrfs file system it must load the btrfs module, thus issues caused.
The default partitioning to date has always been for /boot to be ext4 (at least on the workstation edition) so loading that module with grub before the kernel is loaded is unnecessary.
I would recommend that you at least consider moving /boot to an ext4 partition to simplify the process. This thread shows how that can be accomplished relatively easily.
There is no problem with grub. It works fine. The kernels are read and loaded properly.
The problem is with dnf/rpm. It looks like the rpm scriptlet is calling grubby (not grub) and that throws an error during installation/update BUT the kernels work properly despite of that.
I am not looking to migrate /boot to ext4 since it IS actually working fine. What I’m looking for is a way to figure out what exactly is going on with the rpm scriptlet and grubby so I can properly report the bug.