F35 Server ARM edition Ethernet port not found

I’ve been using ARM server edition with R Pi 3B+ for two months. I use it for a backup server, which is set for an automatic system update every Sunday.
What happened is;

  • Since the last server update (26 Dec), the Ethernet port is down. LED not blinking.
  • ssh into the server with the headless setup is not possible.
    sh: connect to host IP address port 22: No route to host

Basic troubleshooting steps I followed;

  • I checked with a different device and the router is okay.
  • Having connected a monitor and keyboard, I rebooted the server and the first line of the bootup screen shows ‘No Ethernet is found’.
  • Using a different bootable micro SD card (not a backup image), the server is booting okay as well as the Ethernet port blinking. As the host is changed, I’m not allowed to log in.

I checked with the Raspberry Pi forum about potential hardware issues, but there are no relevant articles.

Any suggestions will be much appreciated.

Debian is far behind with kernel versions. So it is possible that an update made changes where Debian is just aware in some weeks or month.

Have a look if you can install the program inxi
Then make us an:
inxi -Fzx in terminal and post the output as </> Preformatted text here.
So we do see what hardware got detected and what driver is used.

Which release of Fedora Linux?
Are you using the armv7 or the aarch64 image?

It is armv7 raw image.

The original boot disk (MIcroSD Card) stops booting in the middle, so I don’t know how to execute those programs and commands.

Initial boot screen showed ‘hit any key to stop autoboot’,
Keystroke to the USB keyboard does not work during boot time.
See the screen shot.
I can’t enter rootpassword.

Could be related to this? 2035802 – libzstd-1.5.1-1.fc35.s390x completely breaks s390

1 Like

Redo the image on the sd card. The error is initramfs is not finding the boot image …
“Warning: /dev/fedora_fedora/root does not exist”
"Warning: /dev/mapper/fedora_fedora-root does not exist:

Stephen

Does redo the image mean complete reinstallation or restoring the boot image (partition??)?

Can you suggest how I can do that with a disk repair utility?

As the keyboard does not work for the USB port of R Pi, recovery of the boot image using my laptop should be the fastest way.

Well, looking at it a bit closer I think maybe your mount point is missing since it is the device that is missing as opposed to the image itself as noted by the warning. What I was meaning is maybe the SD card is in need of being re-imaged. Perhaps as per the instructions at https://docs.fedoraproject.org/en-US/quick-docs/raspberry-pi/#installing-fedora-on-a-raspberry-pi-using-the-fedora-arm-installer_rpi

As the server boot stops in the middle, I can’t run inxi, which can’t check external device if I take out the sd card and run the disk check in my laptop.

I ran fsck in my laptop for the sd card (boot image).

Device Boot Start End Sectors Size Id Type
/dev/sdb1 * 2048 1230847 1228800 600M 6 FAT16
/dev/sdb2 1230848 3327999 2097152 1G 83 Linux
/dev/sdb3 3328000 62521343 59193344 28.2G 8e Linux LVM

$ sudo umount /dev/sdb1
umount: /dev/sda1: not mounted.

$ sudo fsck -y /dev/sdb1
fsck from util-linux 2.37.2
fsck.fat 4.2 (2021-01-31)
/dev/sdb1: 277 files, 7125/153290 clusters

umounted all devices.

$ sudo fsck -a /dev/sdb1
same results

The above was supposed to fix any errors in partitions.

I inserted the sd card into R Pi, but the boot problem remain.

Any suggestions to fix the damaged boot partition will be appreciated.
If there is no other way, then I have to rebuild the server from scratch (hours of config and test gone). A stark reminder to backup.

My bad - I hadn’t backed up the server.

Did you try while booting press esc to select an other kernel version?

No, I didn’t. I wasn’t aware of how to stop autoboot.
I managed to stop by pressing esc and I see U-Boot prompt.

Now the keyboard works.
What commands do you recommend to select other kernel version?
I typed help, but overwhelmed by options.

Pressed esc to go out and esc again. I see grub prompt.

Is there not an older kernel to select ?

When I saw the list of kernels and instantly pressed esc, it doesn’t stop. Instead, I’m greeted with grub prompt.

Have a look if this helps

Ctrl D to stop at kernel options.
I selected earlier version(s)
5.15.10-200: After booting sequence is complete, the screen went blank. I ran ping and ssh but no connection (although Ethernet port is blinking)
5.15.8-200: No booting at all.

Sure. I ran testdisk to revive unbootable partitions, but there was no backup to recover from. Will have to reinstall the server and backup the boot disk after completing the configuration.

I’m not sure, but look like your setup using LVM. Usually fstab need something like /dev/mapper.

Once again not sure how, but you could try first chroot it somewhere to make sure it the installed GNU/Linux are functional then focus on how to mount LVM since I believe it’s need recognized by the bios by something insmod lvm on grub.cfg.

Update:

My bad. Look like I can’t get of rid my bad habit of skim reading.

Since you can see above menu, your UEFI and /boot partition should be fine. Your EFI firmware successfully read the content of /EFI/fedora/grub.cfg that pointing to other grub.cfg in /boot/grub2/grub.cfg.

Then the /boot/grub2/grub.cfg also successfully pointing to list of boot menu from /boot/loader/entries/*.

We could safe to say your /dev/sdb1 and /dev/sdb2 are fine.

Next are your /dev/sdb3. Since this partition using LVM, we need to make sure that fstab (or something equivalent with arm, I don’t know) and kernel parameter are correct. For kernel parameter you could directly check in /boot/loader/entries/. There should some file with each of them contain the boot parameter (same like when on boot list you press e on keyboard to edit on the fly).

Maybe you could share the fstab and one of the recent kernel parameter in /boot/loader/entries also with /boot/grub2/grub.cfg on part start with ### BEGIN /etc/grub.d/10_linux ### to part ### END /etc/grub.d/10_linux ### by mounting it with other machine.

Update:

Bellow are when I messed up with boot parameter with Fedora Workstation on VM:

On your case it mention Warning: /dev/fedora_fedora/root does not exist means something must be changed with your LVM mapping.

Okay, I’ll keep the disk in question for checking later. I have no experience dealing with LVM snapshot yet.
For now, I’m reinstalling the system. I really need to back up the image once the config is done. Thanks.

Maybe this is off-topic, but would it be the case of the typical SD card being corrupted?
When running SBC on an SD card 24/7, how reliable could it be? I’m just weighing pros and cons of a backup plan.

A: Move the system volume group (fedora_fedora) on an SSD drive
B: Move the root file system (/) to an SSD drive and just leave the boot file system (\boot) on the SD card.
C: Just buy an SBC that supports USB boot
D: Mount LVM/backup - maybe not feasible for an unbootable system/less experienced users like me. LVM is auto-created by the ARM installer, but not sure how to use it.

What do you think? Do you have an SD card being a problem during your SBC test and daily use?