Moved / to /dev/sda SSD drives, emergency mode with can't open initiatorname configuration file /etc/iscsi/initiatorname.iscsi

New Fedora 30 install.

  • We have/dev/sda as a hardware RAID 1 with 2 SSDs,/boot and /boot/efi are in it as well.
  • We have 6 HDD’s in a RAID 10 with/dev/sdb and separate partitions for /var and/home (and / before this).
    *I created/dev/sda3 partition and formatted as XFS and a LVM
  • We copied everything from the old root (/dev/mapper/fedora-root), to /dev/mapper/root-lvroot.
  • I updated/etc/fstab with the new /dev/sda3 for root
  • I ran grub2-mkconfig.

Reboots and goes into emergency mode with these errors:

can't  open initiatorname configuration file /etc/iscsi/initiatorname.iscsi
Warning: InitiatorName file  /etc/iscsi/ initiatorname .iscsi  does not exist or does not contain a properly formatted  InitiatorName
*can't open InitiatorAlias* configuration file /etc/iscsi/initiatorname iscsi

Here’s a screenshot of blkid in emergency mode:

Did I miss a step? Should I run grub-install as suggested here?

Here’s what it looks like in gparted:

edit: From systemRescueCD I tried:

    grub2-install --no-floppy --recheck /dev/sda
    grub2-install: error: /usr/lib/grub/x86_64-efi/modinfo.sh doesn't exist. Please specify --target or --directory.

Edit: results of df -H

df -H
Filesystem               Size  Used Avail Use% Mounted on
/dev/mapper/root-lvroot  236G   20G  217G   9% /
/dev/sda1                2.0G  8.2M  2.0G   1% /boot
dev                       68G   20M   68G   1% /dev
/dev/sda2                2.0G  173M  1.7G  10% /boot/efi
run                       68G   97M   68G   1% /run

and when trying to reinstall with dnf:

dnf reinstall grub2-efi grub2-efi-modules shim
Unable to detect release version (use '--releasever' to specify release version)
Fedora Modular $releasever - x86_64             349 kB/s |  53 kB     00:00    
Failed to download metadata for repo 'fedora-modular'
Error: Failed to download metadata for repo 'fedora-modular'

mount
/dev/mapper/root-lvroot on / type xfs (rw,relatime,attr2,inode64,sunit=512,swidth=512,noquota)
/dev/sda1 on /boot type vfat (rw,relatime,fmask=0022,dmask=0022,codepage=437,iocharset=iso8859-1,shortname=mixed,utf8,errors=remount-ro)
dev on /dev type devtmpfs (rw,nosuid,relatime,size=65709296k,nr_inodes=16427324,mode=755)
dev on /dev type devtmpfs (rw,nosuid,relatime,size=65709296k,nr_inodes=16427324,mode=755)
proc on /proc type proc (rw,nosuid,nodev,noexec,relatime)
sys on /sys type sysfs (rw,nosuid,nodev,noexec,relatime)
/dev/sda2 on /boot/efi type ext4 (rw,relatime,stripe=64)
run on /run type tmpfs (rw,nosuid,nodev,relatime,mode=755)

Still in SystemRescueCD:

grub2-mkconfig -o /boot/efi/EFI/fedora/grub.cfg
/usr/bin/grub2-editenv: error: cannot open `/boot/grub2/grubenv.new': No such file or directory.
Generating grub configuration file ...
mkdir: cannot create directory '/var/lib/os-prober/mount': No such file or directory
mkdir: cannot create directory '/var/lib/os-prober/mount': No such file or directory
mkdir: cannot create directory '/var/lib/os-prober/mount': No such file or directory
mkdir: cannot create directory '/var/lib/os-prober/mount': No such file or directory
done

lsblk

NAME            MAJ:MIN RM   SIZE RO TYPE MOUNTPOINT
loop0             7:0    0 788.8M  1 loop 
sda               8:0    0   223G  0 disk 
├─sda1            8:1    0   1.9G  0 part /boot/efi
├─sda2            8:2    0   1.9G  0 part 
└─sda3            8:3    0 219.3G  0 part 
  └─root-lvroot 254:0    0 219.3G  0 lvm  /
sdb               8:16   0   4.9T  0 disk 
└─sdb1            8:17   0   4.6T  0 part 
  ├─fedora-root 254:1    0 791.6G  0 lvm  
  ├─fedora-swap 254:2    0  14.9G  0 lvm  
  ├─fedora-var  254:3    0 186.3G  0 lvm  
  └─fedora-home 254:4    0   3.7T  0 lvm  
sr0              11:0    1   841M  0 rom

Edit: some of the above errors were because/var needed to be mounted. Once I did that I could run the commands but same issue continues.

fsarchiver probe simple
[======DISK======] [=============NAME==============] [====SIZE====] [MAJ] [MIN]
[sda             ] [PERC H740P Adp                 ] [   223.00 GB] [  8] [  0]
[sdb             ] [PERC H740P Adp                 ] [     4.91 TB] [  8] [ 16]
[sr0             ] [Virtual CD                     ] [   841.00 MB] [ 11] [  0]

[=====DEVICE=====] [==FILESYS==] [======LABEL======] [====SIZE====] [MAJ] [MIN] 
[loop0           ] [squashfs   ] [<unknown>        ] [   788.83 MB] [  7] [  0] 
[sda1            ] [vfat       ] [/boot/efi        ] [     1.86 GB] [  8] [  1] 
[sda2            ] [ext4       ] [/boot            ] [     1.86 GB] [  8] [  2] 
[sda3            ] [LVM2_member] [<unknown>        ] [   219.27 GB] [  8] [  3] 
[sdb1            ] [LVM2_member] [<unknown>        ] [     4.61 TB] [  8] [ 17] 
[dm-0            ] [xfs        ] [<unknown>        ] [   219.27 GB] [254] [  0] 
[dm-1            ] [xfs        ] [root             ] [   791.62 GB] [254] [  1] 
[dm-2            ] [swap       ] [swap             ] [    14.90 GB] [254] [  2] 
[dm-3            ] [xfs        ] [<unknown>        ] [   186.27 GB] [254] [  3] 
[dm-4            ] [xfs        ] [<unknown>        ] [     3.64 TB] [254] [  4]

After editing the kernel options during the boot for the old root directory, I changed root to point to /dev/sda3. Now emergency mode errors out with: Failed to mount /sysroot. Is the problem coming from making /dev/sda3 a LVM?

And this command is finding the old root:

grub2-mkconfig -o /boot/efi/EFI/fedora/grub.cfg
Generating grub configuration file ...
Found Fedora 30 (Server Edition) on /dev/mapper/fedora-root

Here’s what I have in /etc/default/grub:

cat /etc/default/grub
GRUB_TIMEOUT=5
GRUB_DISTRIBUTOR="$(sed 's, release .*$,,g' /etc/system-release)"
GRUB_DEFAULT=saved
GRUB_DISABLE_SUBMENU=true
GRUB_TERMINAL_OUTPUT="console"
GRUB_CMDLINE_LINUX="resume=/dev/mapper/fedora-swap rd.lvm.lv=root/lvroot rd.lvm.lv=fedora/swap rhgb quiet audit=0"
GRUB_DISABLE_RECOVERY="true"
GRUB_ENABLE_BLSCFG=true

(It looks like your root LVM PV is on /dev/sda3, not /dev/sdb3, but the boot messages all reference sda3 so I assume /dev/sdb3 was just a typo here.)

That’s where you went wrong. As you said:

  • I created /dev/sda3 partition and formatted as XFS and a LVM
  • We copied everything from the old root (/dev/mapper/fedora-root), to /dev/mapper/root-lvroot.

So your root isn’t /dev/sda3, it’s /dev/mapper/root-lvroot, and that’s how you need to configure GRUB / your system. I’ll read through the rest and try to offer more input, but… that’s the bit you need to fix.

(ETA: The system is failing because, specifically, with the root set to /dev/sda3, the boot process is trying to fsck and mount that partition. But the partition isn’t a filesystem, it’s an LVM2_member that it has no idea how to mount (because it can’t).)

Other than /etc/fstab having the wrong root, it looks like things are pretty OK. You’ve got the GRUB_CMDLINE_LINUX updated properly with rd.lvm.lv=root/lvroot, so that’s correct.

You just need to fix /etc/fstab with the correct root partition, and then generate a new initrd with the updated filesystem layout. The easiest way to do that would probably be to:

  • Boot into the old /dev/mapper/fedora-root from grub, even if you have to manually edit the grub command line to do it. — You may also have to edit the /etc/fstab on that volume from something like SystemRescueCD first, to set the root partition back to /dev/mapper/fedora-root.
  • edit its /etc/fstab again, and set the root partition to /dev/mapper/root-lvroot
  • run this:
    sudo kernel-install add $KRELEASE /lib/modules/$KRELEASE/vmlinuz
    
    where $KRELEASE is the uname -r string corresponding to the latest kernel (5.2.7-200.fc30.x86_64 on my Fedora 30 install). Make sure that kernel’s installed on the /dev/mapper/fedora-root system first, of course.

That’ll update the boot loader entry for that kernel, and regenerate the corresponding initrd using the information in /etc/fstab to define the filesystem layout.

That’s normal/expected, the os-prober lists installs it finds other than the primary one it’s configuring. So, that’s actually an indication that GRUB IS properly considering /dev/mapper/root-lvroot to be your primary install. And then it’s finding an additional Fedora installation at /dev/mapper/fedora-root, because there is one there.

Oops nice catch, updated in original post.

Tried that but now it appears the original root is failing:


59%20PM

Subsequent attempts trying to boot from new root:

13%20PM

Check out the Grub kernel boot options:

17%20PM

I edited the linux line, changed the /dev/dm-1 to/dev/mapper/fedora-root and that brought me back to the old root just fine. I then ran:

And the kernel version is 5.0.9-301.fc30.x86_64.

Rebooted and again Failed to mount /sysroot:
59%20PM

Do I have to run any other commands after kernel-install? Is there something wrong with the way I created the /dev/sda3 partition and subsequent LVM commands?

Here’s /boot/efi/EFI/EFI/fedora/grub.cfg

#
# DO NOT EDIT THIS FILE
#
# It is automatically generated by grub2-mkconfig using templates
# from /etc/grub.d and settings from /etc/default/grub
#

### BEGIN /etc/grub.d/00_header ###
set pager=1

if [ -f ${config_directory}/grubenv ]; then
  load_env -f ${config_directory}/grubenv
elif [ -s $prefix/grubenv ]; then
  load_env
fi
if [ "${next_entry}" ] ; then
   set default="${next_entry}"
   set next_entry=
   save_env next_entry
   set boot_once=true
else
   set default="${saved_entry}"
fi

if [ x"${feature_menuentry_id}" = xy ]; then
  menuentry_id_option="--id"
else
  menuentry_id_option=""
fi

export menuentry_id_option

if [ "${prev_saved_entry}" ]; then
  set saved_entry="${prev_saved_entry}"
  save_env saved_entry
  set prev_saved_entry=
  save_env prev_saved_entry
  set boot_once=true
fi

function savedefault {
  if [ -z "${boot_once}" ]; then
    saved_entry="${chosen}"
    save_env saved_entry
  fi
}

function load_video {
  if [ x$feature_all_video_module = xy ]; then
    insmod all_video
  else
    insmod efi_gop
    insmod efi_uga
    insmod ieee1275_fb
    insmod vbe
    insmod vga
    insmod video_bochs
    insmod video_cirrus
  fi
}

terminal_output console
if [ x$feature_timeout_style = xy ] ; then
  set timeout_style=menu
  set timeout=5
# Fallback normal timeout code in case the timeout_style feature is
# unavailable.
else
  set timeout=5
fi
### END /etc/grub.d/00_header ###

### BEGIN /etc/grub.d/01_users ###
if [ -f ${prefix}/user.cfg ]; then
  source ${prefix}/user.cfg
  if [ -n "${GRUB2_PASSWORD}" ]; then
    set superusers="root"
    export superusers
    password_pbkdf2 root ${GRUB2_PASSWORD}
  fi
fi
### END /etc/grub.d/01_users ###

### BEGIN /etc/grub.d/08_fallback_counting ###
insmod increment
# Check if boot_counter exists and boot_success=0 to activate this behaviour.
if [ -n "${boot_counter}" -a "${boot_success}" = "0" ]; then
  # if countdown has ended, choose to boot rollback deployment,
  # i.e. default=1 on OSTree-based systems.
  if  [ "${boot_counter}" = "0" -o "${boot_counter}" = "-1" ]; then
    set default=1
    set boot_counter=-1
  # otherwise decrement boot_counter
  else
    decrement boot_counter
  fi
  save_env boot_counter
fi
### END /etc/grub.d/08_fallback_counting ###

### BEGIN /etc/grub.d/10_linux ###

### END /etc/grub.d/10_linux ###

### BEGIN /etc/grub.d/10_reset_boot_success ###
insmod increment
# Hiding the menu is ok if last boot was ok or if this is a first boot attempt to boot the entry
if [ "${boot_success}" = "1" -o "${boot_indeterminate}" = "1" ]; then
  set menu_hide_ok=1
else
  set menu_hide_ok=0 
fi
# Reset boot_indeterminate after a successful boot, increment otherwise
if [ "${boot_success}" = "1" ] ; then
  set boot_indeterminate=0
else
  increment boot_indeterminate
fi
# Reset boot_success for current boot 
set boot_success=0
save_env boot_success boot_indeterminate
### END /etc/grub.d/10_reset_boot_success ###

### BEGIN /etc/grub.d/12_menu_auto_hide ###
if [ x$feature_timeout_style = xy ] ; then
  if [ "${menu_show_once}" ]; then
    unset menu_show_once
    save_env menu_show_once
    set timeout_style=menu
    set timeout=60
  elif [ "${menu_auto_hide}" -a "${menu_hide_ok}" = "1" ]; then
    set orig_timeout_style=${timeout_style}
    set orig_timeout=${timeout}
    if [ "${fastboot}" = "1" ]; then
      # timeout_style=menu + timeout=0 avoids the countdown code keypress check
      set timeout_style=menu
      set timeout=0
    else
      set timeout_style=hidden
      set timeout=1
    fi
  fi
fi
### END /etc/grub.d/12_menu_auto_hide ###

### BEGIN /etc/grub.d/20_linux_xen ###
### END /etc/grub.d/20_linux_xen ###

### BEGIN /etc/grub.d/20_ppc_terminfo ###
### END /etc/grub.d/20_ppc_terminfo ###

### BEGIN /etc/grub.d/30_os-prober ###
menuentry 'Fedora 30 (Server Edition) (on /dev/mapper/fedora-root)' --class fedora --class gnu-linux --class gnu --class os $menuentry_id_option 'osprober-gnulinux-simple-5a16d855-1016-4289-8f00-0ccf5211a59c' {
	insmod part_gpt
	insmod ext2
	set root='hd0,gpt2'
	if [ x$feature_platform_search_hint = xy ]; then
	  search --no-floppy --fs-uuid --set=root --hint-ieee1275='ieee1275//sas/disk@20000,gpt2' --hint-bios=hd0,gpt2 --hint-efi=hd0,gpt2 --hint-baremetal=ahci0,gpt2  6e02c8ac-3f94-4c51-ba02-eabe7a0a4a86
	else
	  search --no-floppy --fs-uuid --set=root 6e02c8ac-3f94-4c51-ba02-eabe7a0a4a86
	fi
	linux /vmlinuz-0-rescue-013a5dc230064052a45158916f1e6688 root=/dev/dm-0
	initrd /initramfs-0-rescue-013a5dc230064052a45158916f1e6688.img
}
submenu 'Advanced options for Fedora 30 (Server Edition) (on /dev/mapper/fedora-root)' $menuentry_id_option 'osprober-gnulinux-advanced-5a16d855-1016-4289-8f00-0ccf5211a59c' {
	menuentry 'Fedora 30 (Server Edition) (on /dev/mapper/fedora-root)' --class gnu-linux --class gnu --class os $menuentry_id_option 'osprober-gnulinux-/vmlinuz-0-rescue-013a5dc230064052a45158916f1e6688--5a16d855-1016-4289-8f00-0ccf5211a59c' {
		insmod part_gpt
		insmod ext2
		set root='hd0,gpt2'
		if [ x$feature_platform_search_hint = xy ]; then
		  search --no-floppy --fs-uuid --set=root --hint-ieee1275='ieee1275//sas/disk@20000,gpt2' --hint-bios=hd0,gpt2 --hint-efi=hd0,gpt2 --hint-baremetal=ahci0,gpt2  6e02c8ac-3f94-4c51-ba02-eabe7a0a4a86
		else
		  search --no-floppy --fs-uuid --set=root 6e02c8ac-3f94-4c51-ba02-eabe7a0a4a86
		fi
		linux /vmlinuz-0-rescue-013a5dc230064052a45158916f1e6688 root=/dev/dm-0
		initrd /initramfs-0-rescue-013a5dc230064052a45158916f1e6688.img
	}
	menuentry 'Fedora 30 (Server Edition) (on /dev/mapper/fedora-root)' --class gnu-linux --class gnu --class os $menuentry_id_option 'osprober-gnulinux-/vmlinuz-5.0.9-301.fc30.x86_64--5a16d855-1016-4289-8f00-0ccf5211a59c' {
		insmod part_gpt
		insmod ext2
		set root='hd0,gpt2'
		if [ x$feature_platform_search_hint = xy ]; then
		  search --no-floppy --fs-uuid --set=root --hint-ieee1275='ieee1275//sas/disk@20000,gpt2' --hint-bios=hd0,gpt2 --hint-efi=hd0,gpt2 --hint-baremetal=ahci0,gpt2  6e02c8ac-3f94-4c51-ba02-eabe7a0a4a86
		else
		  search --no-floppy --fs-uuid --set=root 6e02c8ac-3f94-4c51-ba02-eabe7a0a4a86
		fi
		linux /vmlinuz-5.0.9-301.fc30.x86_64 root=/dev/dm-0
		initrd /initramfs-5.0.9-301.fc30.x86_64.img
	}
	menuentry 'Fedora 30 (Server Edition) (on /dev/mapper/fedora-root)' --class gnu-linux --class gnu --class os $menuentry_id_option 'osprober-gnulinux-/vmlinuz-5.2.7-200.fc30.x86_64--5a16d855-1016-4289-8f00-0ccf5211a59c' {
		insmod part_gpt
		insmod ext2
		set root='hd0,gpt2'
		if [ x$feature_platform_search_hint = xy ]; then
		  search --no-floppy --fs-uuid --set=root --hint-ieee1275='ieee1275//sas/disk@20000,gpt2' --hint-bios=hd0,gpt2 --hint-efi=hd0,gpt2 --hint-baremetal=ahci0,gpt2  6e02c8ac-3f94-4c51-ba02-eabe7a0a4a86
		else
		  search --no-floppy --fs-uuid --set=root 6e02c8ac-3f94-4c51-ba02-eabe7a0a4a86
		fi
		linux /vmlinuz-5.2.7-200.fc30.x86_64 root=/dev/dm-0
		initrd /initramfs-5.2.7-200.fc30.x86_64.img
	}
}

# Other OS found, undo autohiding of menu unless menu_auto_hide=2
if [ "${orig_timeout_style}" -a "${menu_auto_hide}" != "2" ]; then
  set timeout_style=${orig_timeout_style}
  set timeout=${orig_timeout}
fi
### END /etc/grub.d/30_os-prober ###

### BEGIN /etc/grub.d/30_uefi-firmware ###
### END /etc/grub.d/30_uefi-firmware ###

### BEGIN /etc/grub.d/40_custom ###
# This file provides an easy way to add custom menu entries.  Simply type the
# menu entries you want to add after this comment.  Be careful not to change
# the 'exec tail' line above.
### END /etc/grub.d/40_custom ###

### BEGIN /etc/grub.d/41_custom ###
if [ -f  ${config_directory}/custom.cfg ]; then
  source ${config_directory}/custom.cfg
elif [ -z "${config_directory}" -a -f  $prefix/custom.cfg ]; then
  source $prefix/custom.cfg;
fi
### END /etc/grub.d/41_custom ###

I’ll be honest, I don’t know that I’ve ever successfully swapped root partitions without ending up booted from recovery media at some point. It’s hard to set up a running session on a partition that’s separate from the booted one, with grub. But, going back to what you tried…

That’s your recovery image, though. That’ll work for booting, but… well, to be honest I’m not sure whether it’ll work for fixing this, it might make things worse. If nothing else it’ll complicate the process some.

But you might have that kernel installed, at least on your old root, based on your grub menuentries. If you’ve only upgraded the kernel once or twice, it’s possible. I know I don’t.

You can check it with sudo dnf -C list kernel --showduplicates from a running system. 5.0.9-301.fc30 was upgraded away long ago, on mine. Here’s what I get:

$ sudo dnf -C list kernel --showduplicates
Installed Packages
kernel.x86_64                      5.1.20-300.fc30                      @updates
kernel.x86_64                      5.2.5-200.fc30                       @updates
kernel.x86_64                      5.2.7-200.fc30                       @updates
Available Packages
kernel.x86_64                      5.0.9-301.fc30                       fedora  
kernel.x86_64                      5.2.8-200.fc30                       updates 

I need to update to the latest release 5.2.8-200.fc30, looks like. Fedora 30 originally shipped with 5.0.9-301.fc30 which explains why it’s available. But that kernel’s not installed on my system, so if I try to boot my root partition off it, that won’t work.

(None of the right modules will be installed, which means nothing that requires a kernel module will work — including mounting /boot/efi, which requires the vfat module. I found that out to my surprise last night, testing on my own laptop. Perhaps I have an old recovery, and that’s been corrected by now, because it made my recovery useless for… well… recovery.)

The only kernel-install add lines that would work for setting up my root partition would currently be 5.1.20-300.fc30, 5.2.5-200.fc30, or 5.2.7-200.fc30.

Grub’s os-prober has automatically been setting up entries to do that, with your current kernels, and it looks like 5.0.9-301.fc30 might be among them (assuming it checked that, and didn’t just blindly copy out-of-date enties). Those are the entries like this one:

The confusing thing is that it’s naming all of the entries the same (see this other thread about that), so you might want to manually edit those entries to read something like:

menuentry 'Fedora 30 (Server Edition) (5.0.9-301.fc30 on /dev/mapper/fedora-root)' ...
    ...
    linux /vmlinuz-5.0.9-301.fc30.x86_64 root=/dev/dm-0
    initrd /initramfs-5.0.9-301.fc30.x86_64.img
}
menuentry 'Fedora 30 (Server Edition) (5.2.7-200.fc30 on /dev/mapper/fedora-root)' ...
    ...
    linux /vmlinuz-5.2.7-200.fc30.x86_64 root=/dev/dm-0
    initrd /initramfs-5.2.7-200.fc30.x86_64.img
}

(If you run grub2-mkconfig after doing that, it’ll probably undo it, so if you do run it be prepared to edit those entry titles again.)

That way, at least you’ll know what you’re booting into, and have a fighting chance of it working. If you can boot into your old image like THAT, so that you have a working system running as you expect, then here’s what I’d do:

sudo mkdir /newroot
sudo mount -t ext4 /dev/mapper/root-lvroot /newroot

Then you can sudo vim /newroot/etc/fstab to edit the fstab on your new root volume, to make sure it’s correct (that it lists the / partition as /dev/mapper/root-lvroot). You can also check what kernel versions are available by looking in /newroot/lib/modules/. And if there are any problems with that volume, you’ll know when it tries to mount it.

After you’ve checked that, sudo umount /newroot, and just reboot to grub again without trying any more kernel-add or grub commands. Pick a boot entry that corresponds to one of the kernels you know are installed on /dev/mapper/root-lvroot (aka /newroot), and one-time edit the linux line to use root=/dev/mapper/root-lvroot. If it successfully boots into the new root volume, then you can use grub2-mkconfig and kernel-install add with your latest available kernel release, to set up boot entries that default to that volume.

1 Like