Ask Your Question
3

Boot to new kernel hangs after distro upgrade 25 to 26 to 27 to 28.

asked 2018-09-18 13:04:13 -0600

ecjbosu gravatar image

I could really use some help. I started last Friday to periodic system update on all my Fedora computers at home. I have 7: 3 laptops, 2 workstations, and 2 servers.

I do this every 1 to 2 years upgrading through EOL using dnf system-upgrade followed by dnf ... reboot or previously dnf disto-sync ...

My issue happened on one of my servers upgrading 25 to 26. After running the dnf system-update download and dnf reboot to install the downloaded packages, the system would not boot to the new kernel. It just hung on reached basic target. I could, can get no logs, etc since this is before the system get to a sysreq or vconsole state.

All the other upgrades, but the other server, were completed without issues. Just as documented:) Upgraded all from Fedora 26 to 28.

The 25 to 26 upgraded server could be booted using the 25 kernel from the grum menu, and could be booted into rescue. I then reran the dnf system-update using releasever = 27, followed by the dnf reboot and install. This also failed to boot the new Fedora 27 kernel, but 25 still booted. I tried upgrading to Fedora 28 using the same fully documented steps on the fedora wiki. This time though using dnf distro-sync followed by the grub2-mkconfig ... This still would not boot the new kernel, still hangs at reached basic target. I tried a reinstall of all packages using a dnf ... $(rpm ...) command found online. Thanks for the post. I will update this with the reference later.

The only boot issue seen through all of this was udev for complete device (systemd-udev-settle) failed.

I finally found the new interactive boot command to put on the kernal line in grub2. I executed and interactive boot and stepped through the the boot sequece. I received the syssted-udev-settle failure and then the boot stalled on systemd-tmpfiles .... I think from systemd-tmpfiles-setup.service. It just sits there for hours if I let. It required a hard shutdown reboot, ctlr-alt-del does not reboot.

The two issue in the logs when booted to Fedora 25 kernel are device-mapper: ioctl: ioctl interface mismatch (this occurs ever time a drive is access, I think), multipathd[32215]: Cannot re-arm event polling: Invalid argument, and device-mapper: multipath: blkgetrequest() returned -12 - requeuing.

I really appreciate any and all help. I am not sure what more I can provide or need to provide.

Thanks

edit retag flag offensive close merge delete

Comments

I have an update. I tried disabling the systemd-tmpfiles-setup to no avail. But, when I hit ctl-alt-del I get a Start job is running for dev-mapper-XXXXX\x2droot.device with a time clock and unlimited time.

If this means helps.

Thanks

ecjbosu gravatar imageecjbosu ( 2018-09-18 19:16:03 -0600 )edit

I am not sure, but if "basic target" enables multi-user, you can type Ctrl-Alt-F3 and get a console terminal. Then you could dmesg to see what last happened.

wallyk gravatar imagewallyk ( 2018-09-18 19:50:56 -0600 )edit
2

Please post a comment as a comment to your question, not as an answer.

genodeftest gravatar imagegenodeftest ( 2018-09-19 02:19:06 -0600 )edit
1

I would recommend doing a fresh installation of Fedora. And for the future I recommend installing updates earlier.

genodeftest gravatar imagegenodeftest ( 2018-09-19 02:20:30 -0600 )edit

At the reach basic target in the boot sequence, the virtual console, ctrl-Alt-F3, is not available for me.

Yes, I probably should update faster, but these are my home servers behind good firewalls.

Doing a fresh install is very difficult due to the data I have saved on it and it being our NAS for the home unless I can fresh install without wiping out my data?

ecjbosu gravatar imageecjbosu ( 2018-09-19 09:34:26 -0600 )edit

6 Answers

Sort by ยป oldest newest most voted
1

answered 2018-09-23 17:11:35 -0600

ecjbosu gravatar image

I have found the problem but no solution. A new rebuild with a kernel > 4.14 is not possible with my vintage hardware.

I am running a HP Proliant with P400 SA. This uses the cciss device driver. cciss was superceded by hpsa as of kernel 4.14> for the P400 disk array. F25 that still works in kernel 4.13. Go figure.

hpsa renames all disk devices following the sd, sg, .. format from the dm, mpath, ... format of cciss. So my grub kernel options and fstab will not let the newer kernels boot. I tried removing the devices from the kernel options. This booted to a rescue shell and still vconsole and udev device initialization failed. I did not comment out fstab of my old dev paths, because I do not have the guts if I can't boot back to F25.

http://www.linuxhowtos.org/manpages/4... notes the obsolescence of cciss. Elrepo.org has kmod-cciss for Redhat but it does not work with Fedora.
http://cciss.sourceforge.net/#downloads has the source rpms, so I think I have to brush up on dkms to get add the driver to the kernel. Any feedback on this would be greatly appreciated.

I am still trying kernel options like hpsasimplemode=1.

Thanks

edit flag offensive delete link more

Comments

http://cciss.sourceforge.net/

From time to time there may be instances in which hardware or driver features are not supported by the cciss driver which comes with your distribution or kernel, because it is too old, or the hardware is too new, etc. In these instances you may wish to try the source RPMs or tarballs provided here.

Pick a working machine with F28 and compile RPMs there.

ed209 gravatar imageed209 ( 2018-09-23 17:25:18 -0600 )edit

hpsa_simple_mode=1

ecjbosu gravatar imageecjbosu ( 2018-09-24 08:31:49 -0600 )edit

I tried, on installing the source I received this error root@FSEAL-LT1 software# rpm -iV cpq_cciss-4.6.20-22.src.rpm missing /root/rpmbuild/SOURCES/cciss-4.6.20-22.tar.gz missing s /root/rpmbuild/SPECS/cciss.spec

Tried searching with no answers.

Note the hpsa driver module supports the P400 Smart array. Something happened from kernel 4.13 to 4.14 that does not allow the boot process to find or see the lv's in the array.

ecjbosu gravatar imageecjbosu ( 2018-09-24 12:49:01 -0600 )edit
0

answered 2018-09-23 05:08:58 -0600

ecjbosu gravatar image

Possible BUG:

I started to move toward rebuilding this Server. Downloaded F28 Server and Workstation net-install images and installed on usb stick.
To test, I tried booting into rescue mode on the Bootable USB images. Both the new boots failed as before with: fails to start udev for complete device initialization and hangs at reached Target system.

I tried booting to do the fresh install on both boot images, same thing happened.

I can still boot to my F25 kernel. I am running a HP Proliant DL 380 G5 with a P400 Smart Array. What changed with the new kernels from F25 that now does not read, recognize, or missing mod, or configuration the LVM in my server?

I am adding this as my own answer instead of a comment for this reason.

edit flag offensive delete link more
0

answered 2018-09-21 12:23:34 -0600

ed209 gravatar image

I don't remember exactly when version upgrades started to be reliable. In your case it failed. May I suggest to reinstall from scratch to F28? You'll be done in less than 1 hour. Further investigations by trial and error will cost you more time than being worth.

Just save /home and /etc in order to keep data and various configurations.

edit flag offensive delete link more

Comments

It will take more than an hour to backup up, and reset. I have 1+ TB of data and NAS on this box running apache, tomcat, svn, mariadb, and more. I have USB, of this, but I am not sure of the mariadb data. I really need to redump it. That will take some time. I scrap financial data.

ecjbosu gravatar imageecjbosu ( 2018-09-22 17:01:07 -0600 )edit
1

Wait. You are telling me you don't game with this computer and yet you don't have a proper backup???

ed209 gravatar imageed209 ( 2018-09-22 18:09:44 -0600 )edit

Yes, while I have years of experience with computers and linux, I am not the best at 'proper backups' like disk images etc. I do have sql data dumps of my DB, and rsync copies of my key data/directories to a USB, along with etc. But it is very un-professional and worse, undocumented but for my cron scripts. This my home server. Many things I have here are to debunk when IT staff said it was to hard to use FOSS.

Also, this server is vintage in computer years. Like if you are driving a 1950's Tucker, a short lived ahead of its time car.

ecjbosu gravatar imageecjbosu ( 2018-09-24 08:28:25 -0600 )edit

You asked when upgrades started being reliable. Well I started with F19, my rescue boot is still F19, on this server and over the 5 or six years I have had this server upgraded with no problems to 20, 21, 22, 23, 24, and then 25. Upgrading from 25 to 26 is when the reliability came into question.

ecjbosu gravatar imageecjbosu ( 2018-09-28 09:24:28 -0600 )edit
0

answered 2018-12-22 03:32:25 -0600

updated 2018-12-22 03:34:18 -0600

[ HP DL 380G5 - Fedora 29 ]

Hi. I also use the same model for testing purposes, but F29 does not support built-in graphics. You can install it by adding the graphics card (PIC-E).

If you add an old graphics or some graphic card, it recognizes it well. It solves the problem of hanging or not working during the initial installation process. This problem can be solved easily because this server automatically selects the operation of the graphics card.

 ASUS EAH6450 512MB SILENT LP [ ASUS B750 PI ]

VGA compatible controller: Advanced Micro Devices, Inc. [AMD/ATI] Cedar [Radeon HD 5000/6000/7350/8350 Series] Audio device: Advanced Micro Devices, Inc. [AMD/ATI] Cedar HDMI Audio [Radeon HD 5400/6300/7300 Series]

p400 has a problem that CentOS7 does not recognize, and it has limited problems with other operating systems and has a lot of problems. However, Fedora works well, so if you solve only graphics problems, you can use it normally.

This article was written in hp dl 380g5 and was written to solve related problems and solve problems that other users will encounter.

written by simmon

edit flag offensive delete link more
0

answered 2018-09-22 09:26:02 -0600

ecjbosu gravatar image

I am adding this in the Answer section for my comment on 2 additional items above due to character limitations My disk structure is if it is any help: [root@FSEALShare ~]# lsblk -l

NAME              MAJ:MIN RM   SIZE RO TYPE  MOUNTPOINT
sda                 8:0    0   1.8T  0 disk  
sda1                8:1    0   1.8T  0 part  /media/usbdisk
sr0                11:0    1  1024M  0 rom   
cciss/c0d0        104:0    0 931.5G  0 disk  
cciss/c0d1        104:16   0 931.5G  0 disk  
mpatha            253:0    0 931.5G  0 mpath 
mpatha1           253:1    0   500M  0 part  /boot
mpatha2           253:2    0 790.5G  0 part  
mpatha3           253:3    0  91.7G  0 part  /var
mpatha4           253:4    0     1K  0 part  
mpatha5           253:5    0  48.8G  0 part  /home
mpathb            253:6    0 931.5G  0 mpath 
mpathb1           253:7    0 931.5G  0 part  
FSEALShare-root   253:8    0    50G  0 lvm   /
FSEALShare-swap   253:9    0   7.9G  0 lvm   [SWAP]
FSEALShare-share  253:10   0 732.4G  0 lvm   /Share
FSEALShare-DBdata 253:11   0 931.6G  0 lvm   /DBdata
edit flag offensive delete link more
0

answered 2018-09-28 22:49:09 -0600

ecjbosu gravatar image

I want to provide a follow up. I updated the P400 firmware and the ILO2 firmware for this DL380 G5. I rebooted for each update just to be sure. P400 Firmware from 4.12 to 7.24

I was working some other recommendations when, I decided to try and boot to a F28 kernel. And I'll be if it didn't boot to the F28 4.18.9-200 kernel. This blew my mind. I did try the 4.18.8 without success.

I am not sure, but it seems that something in the new version P400 controller firmware was needed for the Kernel to see the disks and the partitions. The driver is hpsa since the devices are follow sdX and not ccissX.

I want to thank you all for the assistance. I am not sure if things are fully fixed, but I am at least running on the latest kernel.

Thank you all again.

Joe

edit flag offensive delete link more

Your Answer

Please start posting anonymously - your entry will be published after you log in or create a new account.

Add Answer

Question Tools

1 follower

Stats

Asked: 2018-09-18 13:04:13 -0600

Seen: 341 times

Last updated: Dec 22 '18