Help! Fedora just randomly deleted hours of my work and bricked itself all on its own

I recently saw Fedora getting a lot of recommendations as a more stable distro, so switched over from ubuntu and have been using for about two weeks.

I’ve barely did anything other than install synthing for automatic network backup, and migrate files over, as I had a new job starting this week

Ive been working with a bunch of spreadsheets and large-ish (50MB) text files open at once, and performance has been oddly atrocious vs ubuntu (laggy and jittery scrolling down the text files), and then today I started getting random “filesystem is getting full” messages. For some reason there were THOUSANDS of drafts of the text file I was editing, saved in the .local directory. no idea why.

I deleted these drafts (definitely NOT the working files I was using, I double checked first!!), but it kept getting slower so after about an hour I tried to save the spreadsheets and reboot.

This failed too, saying the file didn’t exist or I ‘didn’t have permission’ …to save my own file in Home!! after hanging for several minutes, I then went to look in home, and found the very spreadsheets I was working on were totally gone. any time i tried to resave the working copy anywhere it wouldnt let me

I was starting to get scared so i checked the desktop syncthing was set to backup to, and found for some reason NOTHING from the fedora machine had synced for like 3 days! no idea why it stopped (worked fine at first), because when i got back from the other room the laptop had totally hung/frozen. I waited for an hour but had to hard restart it.

Now when i try and boot, after the encryption password, it hangs. if i hit Esc, I see tons of [FAILED] and “Stopped Xyzabd.service - Userspace Out-Of-Memory Killer.” messages. then it just stops with a blinking cursor but i cant type anything

what is going on??? i literally only installed syncthing and a couple of other apps from the repos which are the same versiosn as i use on ubuntu. how has it broken itself so badly just editing a spreadhsheet? it boots into win fine so its not an ssd/ram problem, but ubuntu and fedora live usbs wont boot now either so i cant open the LUKS partition

i need to recover this ASAP because the the backup not working theres like 4 days work lost on there and i CANT replicate that data!!!..assuming it didnt actually literally delete the file itself already like it seemed to

HELP!

3 Likes

Hi @timmithy , welcome to the forum.

That sounds like quite a weird issue, and should not really be something at an OS level. From your description, it seems like something is creating lots of temporary files which has filled up your disk.

So—what apps/tools were you using when this happened? What tool were you using to edit the text files? Is it configured to periodically save versions of the file for backup/restore purposes, for example?

The system (any OS) will keep getting slower as memory fills up. So here, it seems like the app you were using was creating multiple copies of your file and storing it in memory—causing your memory to fill up.

We’ll have to look at the system logs to figure out why syncthing stopped working. I’ve got it running on 3 different machines here, and it seems to be working fine. It could be another issue of your system memory/disk space filling up and causing syncthing to stop.

I was going to suggest you use live media to at least copy your files over, but if they won’t boot, I don’t know what to suggest. Nothing that’s happened on your system should affect the ability of the machine to boot from live media—or so I thought.

(PS: I’ve edited your post to remove the use of profanity. I totally understand your frustration here, but we don’t permit such language on the forum under any circumstances.)

1 Like

the built in spreadsheet app (calc) and text editor. thunderbird emails was open too and brave web brower, all from the repos

how do I do that? it wont let me type anything when it stops booting

thanks

ps ive run the bios hardware diagnostics and everythings fine, it also boots a windows recovery usb fine

2 Likes

Could the refusal to boot a linux USB or from grub, yet allow booting a windows USB, be bios and possibly tpm related? What version of windows is on the USB? Do you get the grub menu?

2 Likes

I see the “text editor” has a preference to “restore session”, so perhaps it saves periodic snapshots of the files. That’s all I can really think of that may cause lots of files filling up the disk/memory.

Could you take a picture of this perhaps so we can see what services these are?

Can you try to boot into a non UI mode, like single user mode?

I don’t have any ideas on why live usbs won’t boot. That should really be a BIOS issue and not have anything to do with the installed OS…

1 Like

it does boot, it gets to the encryption password screen, and it registers that just fine and starts trying to boot, so thats well past GRUB.

the problem seems to be something to do with all the FAILED messages

1 Like

1 Like

I get this

2 Likes

if i then hit enter, it produces this

2 Likes

You should be able to get into the root shell at boot by adding rw init=/bin/bash and removing rhgb quiet from the line beginning with ‘linux’ in the grub edit screen

If you can get into the single user root shell then you can give root a password (passwd root in that shell) which would then allow you to log in from the emergency mode screen in the future.

It looks like for some reason the system is failing to mount /boot/efi and that is causing the failure to complete the boot.

The failure to mount /boot/efi should be investigated. If you know which partition that is then you could mount it once in the single user shell and view the content. You could also do a file system check on it before mounting with fsck.vfat <device>. A third check to the cause may be to look at /etc/fstab and verify that entry has not been corrupted which could prevent the mount.

3 Likes

how is it booting to the luks password entry then?

I’ve accidentally wiped efi partitions before and it didnt do this, it couldn’t even boot at all

and how did saving a spreadsheet somehow do this???

2 Likes

How it was caused needs to be solved, but this shows in each of the images you have posted that show boot failure.
Screenshot from 2022-11-17 10-27-35
Both the failed to mount and the depend lines.

It would appear that grub is able to read that partition and start the boot, but the file system fails to mount.

2 Likes

This is indeed a mystery. Neither the spreadsheet nor the text editor should have permissions to write to system directories. It should be a system process. Was an update in progress perhaps when the system hung? Such a transaction being interrupted can leave the system in an inconsistent state—if it were updating the kernel and associated boot entries, for example, it could’ve affected the /efi partition. (Does that make sense, @computersavvy )

1 Like

But it wouldn’t have affected the bootloader. So if that is the only problem, then picking the previous kernel should still work fine.

2 Likes

Not really. The fact that the file system fails to mount even though the efi partition is readable would more point to something else being corrupted. BIOS is able to read the efi partition and start the boot so that does not seem corrupt. Maybe the /boot/efi mountpoint entry in /boot got corrupted? Maybe /etc/fstab got corrupted? (though that is much less likely).

The OP really needs to be able to boot to a live image or to the rescue mode / single user mode to solve this.

I agree that it may be possible to use an older kernel, but if the mount point on /boot is corrupt even that may not work.

3 Likes

I tired this, but it didn’t work, output below:

1 Like

booting with the “single” option again does this

1 Like

ive been googling a bit and am startting to think this is a BTRFS problem. theres heaps of people who seem to have lost whole drives in the past when BTRFS gets full, and cant seem to manage it for some reason

i dont think it has anything to do with syncthing - the folder withthe huge pile of drafts text files was /home/[user]/.local/share/org.gnomeTextEditor/drafts, and synthing wasn’t set to back it up at all (only the documents folder) and that was only set to go one-way. also - syncthing can’t duplicate files onto the same host machine

i think the text editor has filled the drive up with draft files by saving thousands of ‘restore sessions’ the big text files I was looking at (i wasnt saving those files and didn’t care, they’re the source data and i can get new copies of that.

But once the drive was full, BTRFS broke even though i deleted the text files, and thats why my spreadsheets ‘diappeared’ - trying to be saved on a drive that BTRFS couldnt empty/manage.

then when it froze and i restarted…the who filesystem has been ruined by btrfs so it cant mount /boot/efi

thats my theory so far.

Kindof unreal this distro breaks itself so badly, so easily tbh

1 Like

I think the more serious problem is the failure to start the dbus daemon. Without it a lot of things doesn’t work.

1 Like

Hi @chrismurphy : sorry for tagging you here, but you are the btrfs expert here :slight_smile:

Does this sound like something btrfs related? If it is, any thoughts on a fix/workaround? Thanks very much.