`earlyoom` systemd service doesn't kill offending application

I use an application that uses a lot of memory, and sometimes that crashes my desktop. So I installed earlyoom. When I test from a terminal, it works as expected:

Terminal 1

$ earlyoom

Terminal 2

$ tail /dev/zero

I see the RAM usage growing, and when it is close to the limit, it gets killed.

However running it as a systemd service, and repeating the same test leads to a desktop crash like before. The systemd service seems to have started correctly, so I’m not sure what’s wrong:

$ journalctl -b -1 -u earlyoom
Feb 10 11:53:36 host systemd[1]: Started Early OOM Daemon.
Feb 10 11:53:36 host earlyoom[955]: earlyoom 1.6.2
Feb 10 11:53:36 host earlyoom[955]: Notifying through D-Bus
Feb 10 11:53:36 host earlyoom[955]: Preferring to kill process names that match regex '^(Web Content|Sparrow|Bisq)$'
Feb 10 11:53:36 host earlyoom[955]: Will avoid killing process names that match regex '^(dnf|packagekitd|gnome-shell|...)$'
Feb 10 11:53:36 host earlyoom[955]: mem total: 13830 MiB, swap total: 8191 MiB
Feb 10 11:53:36 host earlyoom[955]: sending SIGTERM when mem <=  3.62% and swap <= 10.00%,
Feb 10 11:53:36 host earlyoom[955]:         SIGKILL when mem <=  1.81% and swap <=  5.00%
Feb 10 13:16:44 host earlyoom[955]: mem avail:    76 of 13830 MiB ( 0.55%), swap free:  754 of 8191 MiB ( 9.21%)
Feb 10 13:16:44 host earlyoom[955]: low memory! at or below SIGTERM limits: mem  3.62%, swap 10.00%
Feb 10 13:16:44 host earlyoom[955]: memory situation has recovered while selecting victim

This is from one of the tests. Not all of them have the last 2 lines: “low memory! …”. Is it possible the tail /dev/zero test leads to oom too fast for earlyoom to respond? How come it works when I do it interactively? Thoughts?

Not looked into this in detail, but EarlyOOM and the systemd OOMD are two different things:

https://fedoraproject.org/wiki/Changes/EnableSystemdOomd

(See the upgrade/compatibility impact section)

So perhaps, they’re configured with different settings?

1 Like

Thanks! This could be it, I didn’t expect earlyoom to be incompatible with systemd-oomd, now that I think about it, makes sense. I’ll report back what I find :slight_smile:

1 Like

Okay, it seems there’s no definitive way to ensure they are compatible with each other. systemd-oomd uses “pressure metrics” from the kernel to make its decisions[1], whereas earlyoom uses a more straightforward resource utilisation metric like, X% of memory/swap is in use.

All user sessions in Fedora by default sets ManagedOOMMemoryPressureLimit=50% (mentioned on the wiki page linked by @ankursinha).

$ oomctl 
Dry Run: no
Swap Used Limit: 90.00%
Default Memory Pressure Limit: 60.00%
Default Memory Pressure Duration: 20s
System Context:
        Memory: Used: 13.2G Total: 13.5G
        Swap: Used: 275.7M Total: 7.9G
Swap Monitored CGroups:
        Path: /
                Swap Usage: (see System Context)
Memory Pressure Monitored CGroups:
        Path: /user.slice/user-1000.slice/user@1000.service
                Memory Pressure Limit: 50.00%
                Pressure: Avg10: 0.60 Avg60: 0.40 Avg300: 0.15 Total: 25s
                Current Memory Usage: 5.9G
                Memory Min: 0B
                Memory Low: 0B
                Pgscan: 1007754
                Last Pgscan: 1007754

So after a bit of trial and error, I settled on the following setting for earlyoom:

mem total: 13830 MiB, swap total: 8191 MiB
SIGTERM when mem <= 10.00% and swap <= 20.00%,
SIGKILL when mem <=  5.00% and swap <= 10.00%

  1. IIUC, systemd uses the “full” metric - man oomd.conf says “… in which all tasks in the control group were delayed.” ↩︎

1 Like