Hi,
like in the title. When I have unity and rider opened in the background and will go to another workspace for like 5 minutes, when I come back both apps are not there anymore and I need to open it again. Which is strange. Also when I was checking journalctl I didn’t see any error that could be the reason. Also it only happens for apps that are heavy on resources.
Anyone has an idea what’s happening?
Edit:
ram usage is 13,6 / 16 GB
It could be the reason, but it only happens on fedora. On every other distro I tried I never encountered this issue.
This is for sure a systemd-oomd action. A process should not claim for long time a huge amount of memory. See man systemd-oomd and oomd.conf. Simplest check is “systemctl stop systemd-oomd” and “systemctl disable systemd-oomd”. (where of course the protection against out-of-memory situations is lost) A second thing to look at in Fedora is zram swap. Part of the memory is used as compressed swap space, reducing the amount of swap to disk. This might or might not be good in case of programs using a lot of (resident?) memory. Looking to the amount of used swap and adding a swap file (partition not needed) when necessary is also a good advice, see above.
May 30 17:12:27 ADRIAN-OMEN systemd[1]: Starting systemd-oomd.service - Userspace Out-Of-Memory (OOM) Killer...
May 30 17:12:27 ADRIAN-OMEN systemd[1]: Started systemd-oomd.service - Userspace Out-Of-Memory (OOM) Killer.
May 31 19:19:02 ADRIAN-OMEN systemd-oomd[836]: Killed /user.slice/user-1000.slice/user@1000.service/app.slice/app-gnome-unityhub-135516.scope due to memory used (15957831680) / total (16162631680) and swap used (7735545856) / total (8589930496) being more than 90.00%
May 31 21:17:39 ADRIAN-OMEN systemd-oomd[836]: Killed /user.slice/user-1000.slice/user@1000.service/app.slice/app-gnome-unityhub-341140.scope due to memory used (15936348160) / total (16162631680) and swap used (7750955008) / total (8589930496) being more than 90.00%
It really is. It’s interesting as other systems, for example ubuntu, that uses early-oom never killed my app. I’ll try creating swap file. Thanks for quick answers.
I have same problem with OOM Killer. It oftet kill some of processes. But I run several application, as Firefox, PhPStorm, several nodejs processes.
journalctl -f -u systemd-oomd
лип 18 14:11:50 localhost.localdomain systemd-oomd[957]: Killed /system.slice/docker-7b3628b5645bf3ae33e4a14726fc0287b84bbf088bb18b8e86f38d9a0d429c17.scope due to memory used (16102498304) / total (16439758848) and swap used (7862882304) / total (8589930496) being more than 90.00%
лип 18 14:13:16 localhost.localdomain systemd-oomd[957]: Killed /user.slice/user-1000.slice/user@1000.service/app.slice/app-org.gnome.Terminal.slice/vte-spawn-b8f0e8e4-2b19-4439-8ddb-e7976ba7d72d.scope due to memory used (16199786496) / total (16439758848) and swap used (7878807552) / total (8589930496) being more than 90.00%
лип 18 14:19:30 localhost.localdomain systemd-oomd[957]: Killed /system.slice/docker-7b3628b5645bf3ae33e4a14726fc0287b84bbf088bb18b8e86f38d9a0d429c17.scope due to memory used (15352643584) / total (16439758848) and swap used (7779336192) / total (8589930496) being more than 90.00%
лип 18 15:41:22 localhost.localdomain systemd-oomd[957]: Killed /user.slice/user-1000.slice/user@1000.service/app.slice/app-gnome-jetbrains\x2dphpstorm-54180.scope due to memory used (16174944256) / total (16439758848) and swap used (7746093056) / total (8589930496) being more than 90.00%
лип 18 16:29:07 localhost.localdomain systemd-oomd[957]: Killed /system.slice/docker-7b3628b5645bf3ae33e4a14726fc0287b84bbf088bb18b8e86f38d9a0d429c17.scope due to memory used (16211755008) / total (16439758848) and swap used (7759474688) / total (8589930496) being more than 90.00%
лип 18 17:49:59 localhost.localdomain systemd-oomd[957]: Killed /system.slice/docker-7b3628b5645bf3ae33e4a14726fc0287b84bbf088bb18b8e86f38d9a0d429c17.scope due to memory used (16203161600) / total (16439758848) and swap used (7821754368) / total (8589930496) being more than 90.00%
лип 19 10:36:31 localhost.localdomain systemd-oomd[957]: Killed /user.slice/user-1000.slice/user@1000.service/app.slice/app-gnome-jetbrains\x2dphpstorm-71694.scope due to memory used (16129179648) / total (16439758848) and swap used (7801118720) / total (8589930496) being more than 90.00%
лип 19 11:04:00 localhost.localdomain systemd-oomd[957]: Killed /system.slice/docker-7b3628b5645bf3ae33e4a14726fc0287b84bbf088bb18b8e86f38d9a0d429c17.scope due to memory used (16131563520) / total (16439758848) and swap used (7757705216) / total (8589930496) being more than 90.00%
лип 19 12:30:02 localhost.localdomain systemd-oomd[957]: Killed /user.slice/user-1000.slice/user@1000.service/session.slice/org.gnome.Shell@wayland.service due to memory used (16178880512) / total (16439758848) and swap used (7738769408) / total (8589930496) being more than 90.00%
лип 19 13:16:52 localhost.localdomain systemd-oomd[957]: Killed /system.slice/docker-7b3628b5645bf3ae33e4a14726fc0287b84bbf088bb18b8e86f38d9a0d429c17.scope due to memory used (16203325440) / total (16439758848) and swap used (7739105280) / total (8589930496) being more than 90.00%
Also here free -ght output:
total used free shared buff/cache available
Mem: 15Gi 9,2Gi 693Mi 2,7Gi 5,5Gi 3,1Gi
Swap: 8,0Gi 4,4Gi 3,6Gi
Total: 23Gi 13Gi 4,3Gi
As I understand, I already have swap file cat /proc/swaps:
Filename Type Size Used Priority
/dev/zram0 partition 8388604 4837864 100
So, any idea what I need to setup, to prevent killing needed apps?
If you are limited in physical ram (16GB) and are using it all (15GB) then the ONLY solution is to add swap space. Failure to do so with the safety features like ‘systemd-oomd’ disabled is asking for the system to crash or freeze with no way to predict what happens or what damage may occur to your file systems.
This is not a bug, but a feature of the way your system is operating, and the safety features that are built in.
OTOH, it is your system and you can disable oomd if you wish. Just be aware that you might experience unpleasant consequences by doing so.
If you created a partition as seems to be indicated by the UUID then you should not mount it at a system mount point.
If you actually used mkswap to format it on that partition then the system should see it as swap and it can be activated manually using swapon (man swapon). As I understand it the system actually scans and uses all existing swap spaces without needing any entry in fstab although using fstab is certainly not wrong.
From the man page for fstab the second field should be “none” for swap partitions. Otherwise your entry seems right. I am not sure about the syntax if you are using a swapfile, especially on btrfs.
I’m having a similar issue, but it seems to be a legit kill:
Jun 10 06:17:34 carlos-desktop.lan systemd-oomd[1479]: Killed /user.slice/user-1000.slice/user@1000.service/session.slice/org.gnome.Shell@wayland.service due to memory used (15791357952) / total (16103833600) and swap used (7785259008) / total (8589930496) being more than 90.00%
Then then Jul 7
Killed /user.slice/user-1000.slice/session-2.scope due to memory used (15319617536) / total (15988076544) and swap used (7766499328) / total (8589930496) being more than 90.00%
Then a few days ago:
Killed /user.slice/user-1000.slice/session-2.scope due to memory used (15077998592) / total (15988076544) and swap used (7742668800) / total (8589930496) being more than 90.0%
Those ones on session actually kill my Desktop session, and all the apps that I had opened.
I’m just leaving applications opened, and I don’t turn off my computer for days.
I setup a gnome extension to monitor RAM, to try to get more information when it is happening.
These are the stats just before the last kill (Jul 18 at 9:30):
According to the this instruction record is fstab should be like this
/var/swap/swapfile1 none swap defaults 0 0
But I don’t understand what means in Note this point:
When using Btrfs, do not forget to add the created subvolume to the list as well, and remove the discard,autodefrag options from whichever subvolume gets mounted first (which will control these settings for the whole FS).
A better tutorial (perhaps) is Can I have a swapfile on btrfs?. I have not tested this, but it looks reasonable, although a bit complicated.
If it were my system, I would use gparted to shrink the btrfs partition and then create a new swap partition in the new free space.
To disable the zram swap space, you just uninstall the zram-generator-defaults.noarch package using the dnf command. Use the swapon command the check what swap space your system has.
Also, remember that big files in the /tmp directory will eat up valuable ram space. as /tmp nowadays is a ramdisk file system.