Cannot mount BTRFS subvolume to /home from within LVM

Hi,

A fresh default install of Fedora 33 server on a 372GiB SSD resulted in two partitions, one boot (1 GiB) and one LVM partition (371 GiB).

Within the LVM, a 15 GiB partition was created and formatted with XFS as filesystem containing the entire system.

I would like to use the remaining space as BTRFS partition in order to have some (~4-5) subvolumes in it.
One of those subvolumes (labelled @home) should be mounted to /home but even when doing so manually, the mount-command seems to succeed (exit-code is 0) and for a glimpse of a second i can also see the mounted subvolume being listed by the output of mount before it silently gets unmounted immediately.

What is causing these unmounts and how can one prevent this from happening?

When mounting the same @home-subvol to a differing mountpoint it stays mounted as expected.

When mounting freshly created XFS-volume to /home it also stays mounted.

I suspected cockpit which I had enabled could interfere but the behaviour remained the same after sudo systemctl stop cockpit.service && sudo systemctl stop cockpit.socket.

I am aware that the BTRFS-volume would likely be better located outside of the LVM but since online-shrinking the VG seems impossible I don’t have much of a choice here i guess?

Has anybody observed a similar behaviour?

EDIT: forgot to mention that also when adding a line in /etc/fstab to mount the subvolume to /home, cockpit displays a Inconsistent filesystem mount that won’t go away after rebooting or mounting it through clicking the Mount now button which also won’t mount the volume for more than a very short time.

Check the journal for systemd messages related to unmounting /home. For whatever reason, when systemd thinks something shouldn’t be mounted, it will unmount it. I have heard of this problem, but I haven’t personally experienced it.

It’s a bit simpler to put btrfs directly on a partition. You can shrink/grow btrfs online, but the kernel won’t reload a modified partition map on an in-use drive, you will have to reboot. Whereas btrts on LVM you will be able to do live shrink/grow including the LV

Thanks for checking and replying!

I wouldn’t mind having to reboot after a shrink/grow for the partition map to be reloaded but its a good point. Anyhow, since I don’t know a way of online-shrinking a VG in order to create another partition (outside of the LVM-partiton) I believe there is no easy way towards that scenario.

I’ve taken a peek into the systemd messages via journalctl but nothing really jumped at me here, especially no warnings/errors regarding the search term home.

Digging a bit deeper (not filtering for warnings/errors) and searching for home.mount at least gives a glimpse into the behaviour as I can see that new sessions are being started reported by systemd[1] and in the same second I find entries reading

home.mount: Unit is bound to inactive unit dev-disk-by\x2duuid-

followed by an entry of systemd-logind reading
Session XX logged out. Wainting for processes to exit. (XX being a Number)

and right afterwards
Unmounting /home ...

and then systemd-logind with
Removed session XX. (XX being the same Number as above)

and then
home.mount: Succeeded.
Unmounted /home

The thing is: I don’t see/understand whats causing this or how to prevent it.

Well, at least I made a little progress when after rebooting and having added an entry manually into fstab before for mounting my @home-subvolume to /home indeed worked! YAY!
The next problem was just around the corner when I thought I’d mount my other subvolume out of the same BTRFS-volume to the respecitve mountpoint to other branches of the FHS leading to a sudden disconnect from that system.
Afterwards it came up in emergency mode not even giving me a chance to enter any command. :worried:

I managed to edit the second entry out of /etc/fstab through booting from a live-cd and the system came up afterwards. :sweat_smile:
I wish I had remembered to shrink that VG while I had access to the live system but I was too worried in that moment I guess.
So for the moment it seems as if using BTRFS-subvolumes on Fedora 33 server is not going that well.

It does work for the moment as long as I stick to only one auto-mounted (through fstab) subvolume but I think as soon as I have another on configured the system won’t run any more.
Ironically, when I mount the subvolume later manually, everything seems normal (except cockpit won’t display it in the filesystems-tab of the surrounding LV). :thinking:

one question here:

does LVM have any advantages (apart from live-shrinking/growing) ?

I have an separate ext4-partition as /boot and an btrfs-pool for the remaining space (minus ~10 % for Over-Provisioning)
in that pool I have subvolumes as /home, /home/ron/DATA and /

cat /etc/fstab:

UUID=d01eb8eb-78a7-4648-a9f6-0de57039116c / btrfs subvol=ROOT 0 0
UUID=ac2d3c23-941f-427a-9038-735bdc0a4636 /boot ext4 defaults 1 2
UUID=d01eb8eb-78a7-4648-a9f6-0de57039116c /home btrfs subvol=HOME 0 0
UUID=d01eb8eb-78a7-4648-a9f6-0de57039116c /home/ron/DATA btrfs subvol=DATA 0 0
tmpfs /var/tmp tmpfs defaults,nosuid,nodev,noexec,mode=1777 0 0

does LVM have any advantages (apart from live-shrinking/growing) ?

It may depend on the particular use-case but I don’t see any other advantages.
If I had known that the F33 installer still used XFS within LVM I would probably have manually partitioned the system during installation to avoid using LVM at all as it is not necessary for that machine’s partticular use-case and configuration.

1 Like

Having followed earlier discussion of BTRFS during the last couple months before 33 was released it seems that btrfs works similar to LVM with subvolumes already contained that would be similar to LVs. AFAIK BTRFS and LVM are not yet playing well together since btrfs works similar to lvm native.

The subvolumes will automatically grow as needed and as long as there is space on the disk partition so that eliminates the overhead of manually resizing the LVs when needed.
As I understand it the entire partition is BTRFS. It requires a root subvolume (normally / in the default install) that defines the entire space but only uses a small amount. Then other subvolumes can be added that in reality share the disk but act as a separate partition. It compares to LVM in that the btrfs disk/partition can be viewed as the VG and the subvolumes can be viewed as LVs but without all the overhead to maintain them

1 Like

I don’t know a way of online-shrinking a VG

vgreduce will do it. But it only shrinks by the amount of LVM LE’s unused, i.e. you have to shrink an LV which makes LE’s unused. Shrinking a PV I think is pvresize.

But maybe all you want is to have free space in the VG to grow some other LV, or maybe create a new LV for some other purpose. In that case you don’t need to resize either the PV or VG.

The thing is: I don’t see/understand whats causing this or how to prevent it.

Sounds to me like there’s more than one /home item in fstab and they’re conflicting. Maybe post the fstab?

sudden disconnect from that system.

OK so I have experienced something related to all of this in

Afterwards it came up in emergency mode not even giving me a chance to enter any command

Dunno, I need to see logs. That’s fairly typical if an fstab mount fails, unless that mount point includes the option nofail - sounds to me like assembly problems during startup, which is consistent with the automatic unmounting of /home. There’s a conflict somewhere somehow and it just needs to be found.

Post the fstab and also post cat /proc/cmdline

I’d say the main advantage is LVs (logical volumes) are block devices, and can be formatted with any file system. Whereas Btrfs subvolumes are more like directories.. The Btrfs FAQ lists some distinctions and interactions.

Re: fstab, looks reasonable to me. This is using the “flat” style layout. It’s also reasonable to create subvolumes in ~/ which is the “nested” style layout, in which case you don’t need an fstab entry.

{For system related things, I tend to prefer the “flat” layout. And for things in ~/ I tend to prefer “nested”. Doing a rollback is less complicated with the “flat” style since the things you don’t want rolled back still get mounted in the correct place automatically via fstab entry. Whereas with ~/ you’re not likely to rollback the whole thing, but instead you might copy some things out of snapshots, if you’re taking regular snapshots. It’s a be confusing, but there is a VFS limitation that disallows reflink copies across mounts. So if you try to copy between a mounted /home and a mounted /home/ron/DATA/ a reflink (efficient) copy can’t be created, instead it falls back to full copy. Whereas if it’s a subvolume directly in ~/ just like a directory, you’ll get reflink copying automatically. And in fact it’s useful for mv too because subvolumes are considered separate name spaces, so by default mv actually does reflink copy first and then only once that succeeds will it delete the original. This is all a bit esoteric.}

It’s fine to use them together or it’s a bug. While we definitely want to know about the layout when troubleshooting problems, that’s just because more layers causes more complexity, and it’s important to know the whole story.

It is true that the resource control picture is incomplete if Btrfs is on any device-mapper layer (whether LVM or dm-crypt or mdraid). That is, Btrfs directly on a plain partition, we’re able to do IO isolation very effectively using cgroups v2. That’s still an on-going effort by GNOME and KDE, so I think right now most folks aren’t likely to notice much of a difference even if they do put Btrfs on LVM or dm-crypt; but soon we’ll have a more complete resource control story. The IO isolation is important but it’s only one part of that picture, there’s also memory and CPU isolation so you still get at least those things even if Btrfs is on LVM. And even then, it’ll be workload specific.

2 Likes

Thanks @all for answers/insights

By the way, the Btrfs by default change is only for Workstation and other desktop spins. Server does still use XFS on LVM by default. And ext4 is used by cloud and other images (possibly IoT).

It is possible to use Btrfs with Server, using what I call “Custom Automatic” partitioning. You choose the Custom path, change the partition scheme pop-up menu to Btrfs, and then click the blue text above to create the partitions automatically.

1 Like

LVs can me moved between PVs.
That is, you can move the entire OS to another disk while it is online/working.
It’s really tempting for high availability.

There’re also many other features:
https://en.wikipedia.org/wiki/Logical_Volume_Manager_(Linux)#Features
But those may be difficult to understand for non-server users.

Btrfs can do this as well. I did a quick write up of this here, in the “b” section about cloning.

2 Likes