How to install Fedora 25 in LVM on LUKS without a partition table?

Hello Fedora community,

I am coming here from the Qubes OS forum, seeking some guidance (or at least some answers). My original post, and full description of the situation, is there. I will attempt to summarize here with everything important from there. I do not know if there is anything anyone can do here, since I suspect that this is either a bug in the Anaconda installer or otherwise a limitation of it, but I hope I am wrong.

I want to install Qubes OS R4.0.3. Qubes OS uses the Anaconda installer and its base operating system is Fedora (atop Xen). R4.0.3 uses Fedora 25 and I am not aware of any feasible way to rebase it. I want to install it in LVM on LUKS atop raw data disk sda. By that I mean I want no partitioning, partition table, or filesystem data on the raw sda device, ie no VBR/MBR/GPT table or partitioning information exposed in the clear, so that the entire disk appears as random data. /boot and the LUKS headers will be stored on sdb, a separate USB boot drive.

After manually encrypting, partitioning, and formatting everything from tty2 in the live OS, I proceed to assign mountpoints in “Manual Partitioning”, but sda does not show up under the list of existing partitions and neither do any of the LVM volumes. This is even after decrypting the LUKS container and activating the volume group from tty2. Rescanning disks does not help and in fact causes the volume group to be deactivated, as confirmed by lsblk in tty2, with the following lines in /tmp/anaconda.log and /tmp/lvm.log:

# /tmp/anaconda.log
--
INFO program: Running [XX] lvm vgchange -an qubes_dom0 --config= devices { preferred_names=["^/dev/mapper/", "^/dev/md/", "^/dev/sd"] } log {level=7 file=/tmp/lvm.log syslog=0} ...
INFO program: stdout[XX]:  0 logical volume(s) in volume group "qubes_dom0" now active
INFO program: stderr[XX]:  /usr/sbin/dmeventd: stat failed: No such file or directory
--

# /tmp/lvm.log
--
lvmcmdline.c:1611   DEGRADED MODE. Incomplete RAID LVs will be processed.
libdm-config.c:1064   activation/monitoring not found in config: defaulting to 1
lvmcmdline.c:1617   Processing: vgchange -an qubes_dom0 '--config= devices { preferred_names=["^/dev/mapper/", "^/dev/md/", "^/dev/sd"] } log {level=7 file=/tmp/lvm.log syslog=0}
lvmcmdline.c:1618   Command pid: 2730
lvmcmdline.c:1619   system ID:
lvmcmdline.c:1622   O_DIRECT will be used
libdm-config.c:992   global/locking_type not found in config: defaulting to 1
--
libdm-common.c:1475   qubes_dom0-pool00: Skipping NODE_DEL [trust_udev]
libdm-common.c:1475   qubes_dom0-root: Skipping NODE_DEL [trust_udev]
libdm-common.c:1475   qubes_dom0-pool00-tpool: Skipping NODE_DEL [trust_udev]
libdm-common.c:1475   qubes_dom0-pool00_tdata: Skipping NODE_DEL [trust_udev]
libdm-common.c:1475   qubes_dom0-pool00_tmeta: Skipping NODE_DEL [trust_udev]
vgchange.c:160   Deactivated 3 logical volumes in volume group qubes_dom0
activate/dev_manager.c:755   Getting device info for qubes_dom0-swap [LVM-string1]
ioctl/libdm-iface.c:1838   dm info  LVM-string1 [ noopencount flush ]   [16384] (*1)
activate/dev_manager.c:755   Getting device info for qubes_dom0-pool00 [LVM-string2-pool]
ioctl/libdm-iface.c:1838   dm info  LVM-string2-pool [ noopencount flush ]   [16384] (*1)
ioctl/libdm-iface.c:1838   dm info  LVM-string2 [ noopencount flush ]   [16384] (*1)
activate/dev_manager.c:755   Getting device info for qubes_dom0-root [LVM-string3]
ioctl/libdm-iface.c:1838   dm info  LVM-string3 [ noopencount flush ]   [16384] (*1)
activate/activate.c:1835   Counted 0 active LVs in VG qubes_dom0
vgchange.c:259   0 logical volume(s) in volume group "qubes_dom0" now active
activate/dev_manager.c:755   Getting device info for qubes_dom0-swap [LVM-string1]
ioctl/libdm-iface.c:1838   dm info  LVM-string1 [ noopencount flush ]   [16384] (*1)
activate/dev_manager.c:755   Getting device info for qubes_dom0-pool00 [LVM-string2-pool]
ioctl/libdm-iface.c:1838   dm info  LVM-string2-pool [ noopencount flush ]   [16384] (*1)
ioctl/libdm-iface.c:1838   dm info  LVM-string2 [ noopencount flush ]   [16384] (*1)
activate/dev_manager.c:755   Getting device info for qubes_dom0-root [LVM-string3]
ioctl/libdm-iface.c:1838   dm info  LVM-string3 [ noopencount flush ]   [16384] (*1)
activate/activate.c:1835   Counted 0 active LVs in VG qubes_dom0
mm/memlock.c:562   Unlock: Memlock counters: locked:0 critical:0 daemon:0 suspended:0
activate/fs.c:489   Syncing device names
cache/lvmcache.c:157   Metadata cache: VG qubes_dom0 wiped.
misc/lvm-flock.c:71   Unlocking /run/lock/lvm/V_qubes_dom0
misc/lvm-flock.c:48   _undo_flock /run/lock/lvm/V_qubes_dom0
device/dev-io.c:625   Closed /dev/mapper/luks3
metadata/vg.c:89   Freeing VG qubes_dom0 at 0xXXXXXXXXXXXX
metadata/vg.c:89   Freeing VG qubes_dom0 at 0xYYYYYYYYYYYY
--

Inspecting the logs reveals that Accordion is also “unselecting all items”. All this occurs every disk rescan, even when the volume group is manually reactivated.

When I click “Done” twice, the “Summary of Changes” states that Anaconda recognizes all the encrypting, partitioning, and formatting I have done on sda and wants to “Destroy” every step of it just to insert an MSDOS table.

What can I do about this? For example:

  • Will preventing Anaconda from deactivating the volume group upon disk rescan solve this? How can I do so?
  • Is there something I am doing wrong that I can change to successfully expose the LVM volumes for mountpoint assignment without creating a partition table on sda?
  • Can Anaconda somehow be tricked into thinking sda has a partition table?
  • What can I do to hotfix or modify Anaconda to allow installation on a raw data disk?
  • Is there a way to skip/bypass the “Installation Destination” step and “Begin Installation” after manually assigning mountpoints from tty2?
  • Is a chroot and/or unsquashfs path for installing from the tty2 terminal possible, bypassing Anaconda altogether?
  • Is there some other way to force installation after manually mounting?

The setup I want is possible through other installers (such as Debian) and with minimal bootstrappers that allow you to install from scratch (like pacstrap for Arch and debootstrap for Debian). But it does not seem possible on Fedora, at least through an installer like Anaconda, which does not seem to be able to work on raw devices unless it creates a partition table. If not, I may have to go through the headache of reconfiguring the system post-installation or using an intermediate installation. I want to avoid doing that if I can.

I apologize for the length; this was intended to be a brief summary. If any of you have any advice or suggestions on how I might be able to accomplish the setup I want, preferably during installation from the live OS without any outside tools, I will greatly appreciate it. At this point, I am willing to accept any creative solution, or at least confirmation that this is impossible from installation due to the hardcoded limitations of Anaconda.

Best regards,
John


Expected behavior

Existing LVM volumes on LUKS encrypted raw data disk sda (no partition table, detached LUKS header) are available for mountpoint assignment. Assign mountpoints and proceed with installation.

Actual behavior

sda is not listed, LVM volumes are not listed, even when the LUKS container is decrypted and all volumes activated and available. Rescanning disk deactivates the volume group. The “Summary of Changes”, when the full device tree is opened, includes destroying all LUKS and LVM volumes to create an MSDOS table.

Steps to reproduce (simplified)

  1. Launch Qubes R4.0.3 / Fedora 25 live disk (sr0) with a zeroed main drive (sda) and USB drive (sdb). If you are running the live disk from a USB drive, too, the device labels will be different. This may be reproducible on later release versions of Fedora/Anaconda, as well.
  2. Click “Installation Destination”, select “I will configure partitioning.” and deselect “Encrypt my data.” Click “Done”.
  3. CTRL+ALT+F2 to enter tty2.
  4. cryptsetup LUKS container on sda with LUKS header on sdb. Open the LUKS container.
  5. pvcreate, vgcreate, and lvcreate LVM partitioning inside LUKS container.
  6. Deactivate volume group and close LUKS container.
  7. CTRL+ALT+F6 back into Anaconda GUI.
  8. In “Manual Partitioning”, check under existing partitions. See no sda.
  9. Return tty2. Reopen LUKS container, reactivate volume group, and verify that all logical volumes are open and available. lsblk to see full device block tree.
  10. Return to GUI. Check existing partitions, see no sda.
  11. Rescan disk. Check existing partitions, see no sda.
  12. Return to tty2 and run lsblk. The volume group will be deactivated. grep both /tmp/anaconda.log and /tmp/lvm.log for vgchange to confirm deactivation.
  13. Reactivate volume group.
  14. Repeat steps 10 through 13 until you feel numb inside.
  15. Reactivate volume group.
  16. Click “Done” twice to see “Summary of Changes”: LUKS container and all LVM partitions are listed and scheduled to be destroyed to make way for creating an MSDOS table on sda.
  17. Post to Ask Fedora.
2 Likes

Hello @vgaetera,

Thank you for your reply. But forgive my ignorance: where or how am I to use --onpart? I am new to Fedora and Red Hat systems, so I am very unfamiliar with its details. From my searching, this option appears to be used on part in a Kickstart file. Is that correct? If so, then it is unclear to me how I can use that for my situation. Can the part command be used within an Anaconda live disk, such as from within tty2, or is it only used in a Kickstart file? If the second, then does that mean I need to write a Kickstart file to automate my implementation and pass it to Anaconda? Is this compatible with my existing setup or will it be overwritten?

Relatedly, the Qubes OS documentation on custom installation, which is a slightly modified form of the standard installation, states to run pvcreate on /dev/mapper/luks. Will this be in conflict?

Lastly, what are the implications of this regarding my objective of deniable encryption and avoiding any cleartext data being written to the disk? For example, does this suggestion (running part onpart=sda…) write any partition table or partitioning information to the raw device? If so, is there any way to avoid this, such as by running it on /dev/mapper/luks? If not, then it unfortunately cannot solve my problem without preventing the objectives for which I have the problem in the first place.

Any further elaboration or documentation you can provide this old novice will be greatly appreciated.

Kindest regards,
John

1 Like

I tried different ways but none of them worked, thus I’m afraid Anaconda GUI doesn’t support unpartitioned installation.

Yep, it should be one of the Kickstart options, and the author of the linked thread marked it as solved, so I guess it should work.

If that is not a suitable installation method for you, then your only option is a custom scripted installation like the one you have just mentioned.

Since you plan to utilize full disk encryption, the only possible implications I can think of might be boot issues if something goes wrong with your USB boot drive.

1 Like

Thank you for your attempts and for confirming what I have been experiencing, @vgaetera. It is reassuring to me that I am not the only one who came to this conclusion; perhaps I am not entirely crazy, after all.

Are you aware of any terminal-based method I can use of achieving this from within the live disk? I have never been much of a fan of GUI installers, anyway, so I have no problem with bypassing Anaconda to implement what I want in tty2 if I can. That is how I have done everything thus far, from the drive preparation and custom LUKS encryption to the LVM partitioning and custom formatting. Being able to finish installation from the same terminal would be excellent.

Are Kickstart files able to implement customized setups which are not otherwise possible or accepted through the Anaconda GUI? I am concerned that the custom installation I want cannot be implemented through a Kickstart file because Anaconda will still not recognize it as a valid configuration and refuse to install.

In particular, I worry that Anaconda will refuse to accept installing the GRUB bootloader onto an encrypted /boot partition on USB drive sdb, and that it will refuse to install entirely within the LUKS container on sda without adding any partition table or partitioning to the beginning of the disk, leaving me with an unencrypted or uninstalled /boot and an MSDOS table on sda. The first failure will mean that I have to install without a bootloader and fix it post-installation, or otherwise transfer the unencrypted /boot to an encrypted partition and reconfigure GRUB; and the second failure will prevent me from achieving my goal of deniable full disk-encryption without writing any cleartext data to the disk.

If you or anyone else familiar with Kickstart files can address these concerns of mine by either confirming my suspicions or denying that they will likely be a problem, then perhaps the Kickstart file path is one I should begin pursuing. Otherwise, I anticipate that this may be another dead end and I will have to resort to an intermediate installation route after all.

Regards,
John

1 Like

I have not tried the Kickstart installation yet.

It appears to be well-documented here:

There are some examples in the end of the second link that could be helpful.

Consider to utilize virtualization to simplify testing/experiments.

1 Like

Your continued advice has been illuminating, @vgaetera, and I appreciate you providing it.

After reviewing the documentation, I have concluded that unless I include a %pre script that reproduces all the manual work I have done (and even then it may fail for the reasons I specified above, and because it is unclear how it will handle cryptsetup commands with passphrases and key files), I am very likely unable to implement the setup I want and especially on my existing partitions. This is because I am not seeing any way to decrypt the LUKS container within which these LVM partitions are located through the Kickstart file, especially not with key files; and I am not finding any evidence that Kickstart files can be initiated from within a live disk terminal, such as tty2, only in the boot prompt during initialization, and so I am unable to manually unlock and activate the LUKS containers and LVM partitions prior to initiation. Am I correct in this assessment?

If not, then an explanation of how I can initiate a Kickstart after booting the live disk may be sufficient to continue this path.

But if I am correct, then my only feasible path at this point may be to install Qubes OS (Fedora 25) on an intermediate USB drive (sdc) and then manually copy the installation into sda with cp or rsync because the limitations of Anaconda and Kickstart prevent me from directly installing my setup. If that is the case, then what advice can you or anyone else here give me about proceeding with an intermediate installation path? For example, as far as is understood in Fedora, what files will I need to modify to avoid creating an unbootable, broken, or misconfigured final system? I assume GRUB, crypttab, and fstab will all need to be modified. Is there anything else, such as any locations where remnants of the intermediate USB installation might persist?

All the best,
John

After some further consideration, it may be possible to implement my setup through Anaconda using a Kickstart file that does everything I want, though it may require much testing and/or writing some complex scripts in %pre.

Your advice regarding experimentation within a virtual machine is good and I may do so to test some experimental Kickstart files after I have installed my base system, since USB drives are cheap and Qubes OS is nothing but virtualization within a hypervisor. I intend to be creating a guide on how to implement my setup once I am done and a Kickstart file may be an excellent way to do it, assuming it can handle all my commands. However, at this point I just want to get my base system installed the way I want and having to start over from scratch, just to attempt a Kickstart file that overwrites everything I have done despite already spending a month trying to figure this all out, is very discouraging to me. This is especially the case when I already anticipate that the Kickstart attempts will fail as a result of Anaconda considering my configuration to be invalid and refuse to install.

So, I will continue learning about Kickstart and start crafting one that may work, anyway, but I am more interested in hacking my way in just to get it over with sooner rather than later, whether that means a chroot+unsquashfs path or an intermediate install synchronization one. So, in the meantime, any advice you can give in that direction will be much appreciated. Thanks again for your patience.

Regards,
John