Fedora 36 silently kills apps in the background

Hi,
like in the title. When I have unity and rider opened in the background and will go to another workspace for like 5 minutes, when I come back both apps are not there anymore and I need to open it again. Which is strange. Also when I was checking journalctl I didn’t see any error that could be the reason. Also it only happens for apps that are heavy on resources.
Anyone has an idea what’s happening?

Edit:
ram usage is 13,6 / 16 GB
It could be the reason, but it only happens on fedora. On every other distro I tried I never encountered this issue.

1 Like

My first guess would be the OOM (out of memory) killer that replaced earlyoom in Fedora 34:

https://fedoraproject.org/wiki/Changes/EnableSystemdOomd

4 Likes

Create an additional 8 Gb swap file and see if the issue persists.

Also, you should be able to find oom messages in the journal.

This is for sure a systemd-oomd action. A process should not claim for long time a huge amount of memory. See man systemd-oomd and oomd.conf. Simplest check is “systemctl stop systemd-oomd” and “systemctl disable systemd-oomd”. (where of course the protection against out-of-memory situations is lost) A second thing to look at in Fedora is zram swap. Part of the memory is used as compressed swap space, reducing the amount of swap to disk. This might or might not be good in case of programs using a lot of (resident?) memory. Looking to the amount of used swap and adding a swap file (partition not needed) when necessary is also a good advice, see above.

1 Like

journalctl -f -u systemd-oomd

1 Like
May 30 17:12:27 ADRIAN-OMEN systemd[1]: Starting systemd-oomd.service - Userspace Out-Of-Memory (OOM) Killer...
May 30 17:12:27 ADRIAN-OMEN systemd[1]: Started systemd-oomd.service - Userspace Out-Of-Memory (OOM) Killer.
May 31 19:19:02 ADRIAN-OMEN systemd-oomd[836]: Killed /user.slice/user-1000.slice/user@1000.service/app.slice/app-gnome-unityhub-135516.scope due to memory used (15957831680) / total (16162631680) and swap used (7735545856) / total (8589930496) being more than 90.00%
May 31 21:17:39 ADRIAN-OMEN systemd-oomd[836]: Killed /user.slice/user-1000.slice/user@1000.service/app.slice/app-gnome-unityhub-341140.scope due to memory used (15936348160) / total (16162631680) and swap used (7750955008) / total (8589930496) being more than 90.00%

It really is. It’s interesting as other systems, for example ubuntu, that uses early-oom never killed my app. I’ll try creating swap file. Thanks for quick answers.

2 Likes

A post was split to a new topic: Creating a swap file on btrfs

I have same problem with OOM Killer. It oftet kill some of processes. But I run several application, as Firefox, PhPStorm, several nodejs processes.

journalctl -f -u systemd-oomd
лип 18 14:11:50 localhost.localdomain systemd-oomd[957]: Killed /system.slice/docker-7b3628b5645bf3ae33e4a14726fc0287b84bbf088bb18b8e86f38d9a0d429c17.scope due to memory used (16102498304) / total (16439758848) and swap used (7862882304) / total (8589930496) being more than 90.00%
лип 18 14:13:16 localhost.localdomain systemd-oomd[957]: Killed /user.slice/user-1000.slice/user@1000.service/app.slice/app-org.gnome.Terminal.slice/vte-spawn-b8f0e8e4-2b19-4439-8ddb-e7976ba7d72d.scope due to memory used (16199786496) / total (16439758848) and swap used (7878807552) / total (8589930496) being more than 90.00%
лип 18 14:19:30 localhost.localdomain systemd-oomd[957]: Killed /system.slice/docker-7b3628b5645bf3ae33e4a14726fc0287b84bbf088bb18b8e86f38d9a0d429c17.scope due to memory used (15352643584) / total (16439758848) and swap used (7779336192) / total (8589930496) being more than 90.00%
лип 18 15:41:22 localhost.localdomain systemd-oomd[957]: Killed /user.slice/user-1000.slice/user@1000.service/app.slice/app-gnome-jetbrains\x2dphpstorm-54180.scope due to memory used (16174944256) / total (16439758848) and swap used (7746093056) / total (8589930496) being more than 90.00%
лип 18 16:29:07 localhost.localdomain systemd-oomd[957]: Killed /system.slice/docker-7b3628b5645bf3ae33e4a14726fc0287b84bbf088bb18b8e86f38d9a0d429c17.scope due to memory used (16211755008) / total (16439758848) and swap used (7759474688) / total (8589930496) being more than 90.00%
лип 18 17:49:59 localhost.localdomain systemd-oomd[957]: Killed /system.slice/docker-7b3628b5645bf3ae33e4a14726fc0287b84bbf088bb18b8e86f38d9a0d429c17.scope due to memory used (16203161600) / total (16439758848) and swap used (7821754368) / total (8589930496) being more than 90.00%
лип 19 10:36:31 localhost.localdomain systemd-oomd[957]: Killed /user.slice/user-1000.slice/user@1000.service/app.slice/app-gnome-jetbrains\x2dphpstorm-71694.scope due to memory used (16129179648) / total (16439758848) and swap used (7801118720) / total (8589930496) being more than 90.00%
лип 19 11:04:00 localhost.localdomain systemd-oomd[957]: Killed /system.slice/docker-7b3628b5645bf3ae33e4a14726fc0287b84bbf088bb18b8e86f38d9a0d429c17.scope due to memory used (16131563520) / total (16439758848) and swap used (7757705216) / total (8589930496) being more than 90.00%
лип 19 12:30:02 localhost.localdomain systemd-oomd[957]: Killed /user.slice/user-1000.slice/user@1000.service/session.slice/org.gnome.Shell@wayland.service due to memory used (16178880512) / total (16439758848) and swap used (7738769408) / total (8589930496) being more than 90.00%
лип 19 13:16:52 localhost.localdomain systemd-oomd[957]: Killed /system.slice/docker-7b3628b5645bf3ae33e4a14726fc0287b84bbf088bb18b8e86f38d9a0d429c17.scope due to memory used (16203325440) / total (16439758848) and swap used (7739105280) / total (8589930496) being more than 90.00%

Also here free -ght output:

               total        used        free      shared  buff/cache   available
Mem:            15Gi       9,2Gi       693Mi       2,7Gi       5,5Gi       3,1Gi
Swap:          8,0Gi       4,4Gi       3,6Gi
Total:          23Gi        13Gi       4,3Gi

As I understand, I already have swap file cat /proc/swaps:

Filename				Type		Size		Used		Priority
/dev/zram0                              partition	8388604		4837864		100

So, any idea what I need to setup, to prevent killing needed apps?

Did you read the posts above yours that suggested a larger swap as the fix?

Yes, as I understand main proposition - is create swap file(I think I have already had it) or stop\disable systemd-oomd

According to comments of this bug disable systemd-oomd doesn’t work, so I should mask it.

I want to understand what can I do with this case, exept stop/disable/mask systemd-oomd. And why I need this service enabled if it is source of bug?

For example I run my usual apps:

And after few seconds systemd-oomd kill docker process.

I strongly beleive that with current load my system should be deal well.

For example, when i try stop process I have this warning:

 krp  ~  Projects  sudo systemctl stop systemd-oomd
Warning: Stopping systemd-oomd.service, but it can still be activated by:
  systemd-oomd.socket
 krp  ~  Projects  sudo systemctl status systemd-oomd
● systemd-oomd.service - Userspace Out-Of-Memory (OOM) Killer
     Loaded: loaded (/usr/lib/systemd/system/systemd-oomd.service; disabled; vendor preset: enabled)
     Active: active (running) since Wed 2022-07-20 08:32:39 EEST; 13s ago
TriggeredBy: ● systemd-oomd.socket
       Docs: man:systemd-oomd.service(8)
   Main PID: 170766 (systemd-oomd)
     Status: "Processing requests..."
      Tasks: 1 (limit: 18760)
     Memory: 1.4M (min: 64.0M low: 64.0M)
        CPU: 63ms
     CGroup: /system.slice/systemd-oomd.service
             └─ 170766 /usr/lib/systemd/systemd-oomd

лип 20 08:32:39 localhost.localdomain systemd[1]: Starting systemd-oomd.service - Userspace Out-Of-Memory (OOM) Killer...
лип 20 08:32:39 localhost.localdomain systemd[1]: Started systemd-oomd.service - Userspace Out-Of-Memory (OOM) Killer.

You should stop systemd-oomd.socket as well.

F33 switched to zram vice a regular swap file, Changes/SwapOnZRAM - Fedora Project Wiki
This uses ram not physical disk as swap.

If you need additional swap space it needs to be added. Assuming you’re on btrfs, Creating a swap file on btrfs

The systemd oomd logs are indicating the usage is above 90%
Memory Used: 15.0 Gi
Memory Total: 15.3 Gi

Swap Used: 7.3 Gi
Swap Total: 7.9 Gi

Do you think there is a discrepancy in what oomd is reporting?

You can also look at oomctl to see usage.

If you are limited in physical ram (16GB) and are using it all (15GB) then the ONLY solution is to add swap space. Failure to do so with the safety features like ‘systemd-oomd’ disabled is asking for the system to crash or freeze with no way to predict what happens or what damage may occur to your file systems.

This is not a bug, but a feature of the way your system is operating, and the safety features that are built in.

OTOH, it is your system and you can disable oomd if you wish. Just be aware that you might experience unpleasant consequences by doing so.

Ok, thank you, I also no sure about /etc/fstab

UUID=78384903-c20d-4935-8377-354adf1b0d7f /var/swap/swapfile1     swap    defaults        0 0

Is it correct according to my swapfile path /var/swap/swapfile1 and system UUID 78384903-c20d-4935-8377-354adf1b0d7f ?

If you created a partition as seems to be indicated by the UUID then you should not mount it at a system mount point.

If you actually used mkswap to format it on that partition then the system should see it as swap and it can be activated manually using swapon (man swapon). As I understand it the system actually scans and uses all existing swap spaces without needing any entry in fstab although using fstab is certainly not wrong.

From the man page for fstab the second field should be “none” for swap partitions. Otherwise your entry seems right. I am not sure about the syntax if you are using a swapfile, especially on btrfs.

UUID=78384903-c20d-4935-8377-354adf1b0d7f    none     swap    defaults        0 0

There have been some threads about swapfiles on btrfs, and the fedora docs system admin pages probably give more detail.

I’m having a similar issue, but it seems to be a legit kill:

Jun 10 06:17:34 carlos-desktop.lan systemd-oomd[1479]: Killed /user.slice/user-1000.slice/user@1000.service/session.slice/org.gnome.Shell@wayland.service due to memory used (15791357952) / total (16103833600) and swap used (7785259008) / total (8589930496) being more than 90.00%

Then then Jul 7

Killed /user.slice/user-1000.slice/session-2.scope due to memory used (15319617536) / total (15988076544) and swap used (7766499328) / total (8589930496) being more than 90.00%

Then a few days ago:

Killed /user.slice/user-1000.slice/session-2.scope due to memory used (15077998592) / total (15988076544) and swap used (7742668800) / total (8589930496) being more than 90.0%

Those ones on session actually kill my Desktop session, and all the apps that I had opened.

I’m just leaving applications opened, and I don’t turn off my computer for days.

I setup a gnome extension to monitor RAM, to try to get more information when it is happening.

These are the stats just before the last kill (Jul 18 at 9:30):

sar -3 -r

07:20:08 AM kbmemfree   kbavail kbmemused  %memused kbbuffers  kbcached  kbcommit   %commit  kbactive   kbinact   kbdirty
07:30:00 AM   1321588   1933356  12522312     80.20       268   1116056  24253076    101.05   1868808   2380932     15540
07:40:08 AM    686212   1529476  13055832     83.62       276   1217520  23169896     96.53   1840488   3056944       580
07:50:08 AM   1010364   2049052  12602508     80.72       276   1349716  22051780     91.87   2025120   2482340       448
08:00:08 AM    535164   1475956  12781240     81.86       276   1645732  23005340     95.85   2143412   2821292       248
08:10:08 AM    846204   1980120  12318584     78.90       276   1798128  22570404     94.04   2264568   2717256       484
08:20:08 AM    649000   1708728  12577652     80.56       276   1735376  23161268     96.50   2213468   3038680         4
08:30:08 AM    554212   1461364  13130136     84.10       272   1277184  23207604     96.69   2033256   2290828       420
08:40:08 AM    689420   1435144  13226668     84.71       272   1043856  23210032     96.70   1993784   2093568       144
08:50:08 AM    616904   1664568  12992888     83.22       272   1350184  23217284     96.73   2020596   2361872       340
09:00:08 AM    638340   1370060  13292420     85.13       272   1028860  23419688     97.57   1998996   2141068      1140
09:10:08 AM    639152   1417280  13250924     84.87       272   1069720  23228640     96.78   2043640   2099316       372
09:20:08 AM    670580   1453748  13215612     84.64       272   1074752  23228864     96.78   2050036   2056620       192
09:30:08 AM    634400   1455536  13213044     84.63       272   1112720  23232976     96.80   2093736   2059964       580
09:40:08 AM   4360504   6108044   8893044     56.96      2180   1820168   3601720     15.01    729104   1540968         0
09:50:08 AM   4358036   6106160   8898120     56.99      2180   1820768   3601716     15.01    729604   1541148         0

It smells to me like memory leak in some component in Fedora 36. I’ll keep an eye on it.

According to the this instruction record is fstab should be like this

/var/swap/swapfile1 none  swap              defaults                0 0

But I don’t understand what means in Note this point:

When using Btrfs, do not forget to add the created subvolume to the list as well, and remove the discard,autodefrag options from whichever subvolume gets mounted first (which will control these settings for the whole FS).

What the list? And where these options are?

A better tutorial (perhaps) is Can I have a swapfile on btrfs?. I have not tested this, but it looks reasonable, although a bit complicated.

If it were my system, I would use gparted to shrink the btrfs partition and then create a new swap partition in the new free space.

To disable the zram swap space, you just uninstall the zram-generator-defaults.noarch package using the dnf command. Use the swapon command the check what swap space your system has.

Also, remember that big files in the /tmp directory will eat up valuable ram space. as /tmp nowadays is a ramdisk file system.