rfjakob / earlyoom

earlyoom - Early OOM Daemon for Linux

earlyoom - The Early OOM Daemon

The oom-killer generally has a bad reputation among Linux users. This may be part of the reason Linux invokes it only when it has absolutely no other choice. It will swap out the desktop environment, drop the whole page cache and empty every buffer before it will ultimately kill a process. At least that's what I think that it will do. I have yet to be patient enough to wait for it, sitting in front of an unresponsive system.

This made me and other people wonder if the oom-killer could be configured to step in earlier: reddit r/linux, superuser.com, unix.stackexchange.com.

As it turns out, no, it can't. At least using the in-kernel oom-killer. In the user space, however, we can do whatever we want.

earlyoom wants to be simple and solid. It is written in pure C with no dependencies. An extensive test suite (unit- and integration tests) is written in Go.

What does it do

earlyoom checks the amount of available memory and free swap up to 10 times a second (less often if there is a lot of free memory). By default if both are below 10%, it will kill the largest process (highest oom_score). The percentage value is configurable via command line arguments.

In the free -m output below, the available memory is 2170 MiB and the free swap is 231 MiB.

              total        used        free      shared  buff/cache   available
Mem:           7842        4523         137         841        3182        2170
Swap:          1023         792         231

Why is "available" memory checked as opposed to "free" memory? On a healthy Linux system, "free" memory is supposed to be close to zero, because Linux uses all available physical memory to cache disk access. These caches can be dropped any time the memory is needed for something else.

The "available" memory accounts for that. It sums up all memory that is unused or can be freed immediately.

Note that you need a recent version of free and Linux kernel 3.14+ to see the "available" column. If you have a recent kernel, but an old version of free, you can get the value from grep MemAvailable /proc/meminfo.

When both your available memory and free swap drop below 10% of the total memory available to userspace processes (=total-shared), it will send the SIGTERM signal to the process that uses the most memory in the opinion of the kernel (/proc/*/oom_score).

See also

• nohang, a similar project like earlyoom, written in Python and with additional features and configuration options.
• facebooks's pressure stall information (psi) kernel patches and the accompanying oomd userspace helper. The patches are merged in Linux 4.20.

Why not trigger the kernel oom killer?

earlyoom does not use echo f > /proc/sysrq-trigger because:

In some kernel versions (tested on v4.0.5), triggering the kernel oom killer manually does not work at all. That is, it may only free some graphics memory (that will be allocated immediately again) and not actually kill any process. Here you can see how this looks like on my machine (Intel integrated graphics).

This problem has been fixed in Linux v5.17 (commit f530243a) .

Like the Linux kernel would, earlyoom finds its victim by reading through /proc/*/oom_score.

How much memory does earlyoom use?

About 2 MiB (VmRSS), though only 220 kiB is private memory (RssAnon). The rest is the libc library (RssFile) that is shared with other processes. All memory is locked using mlockall() to make sure earlyoom does not slow down in low memory situations.

Download and compile

Compiling yourself is easy:

git clone https://github.com/rfjakob/earlyoom.git
cd earlyoom
make

Optional: Run the integrated self-tests:

make test

Start earlyoom automatically by registering it as a service:

sudo make install              # systemd
sudo make install-initscript   # non-systemd

Note that for systems with SELinux disabled (Ubuntu 19.04, Debian 9 ...) chcon warnings reporting failure to set the context can be safely ignored.

For Debian 10+ and Ubuntu 18.04+, there's a Debian package:

sudo apt install earlyoom

For Fedora and RHEL 8 with EPEL, there's a Fedora package:

sudo dnf install earlyoom
sudo systemctl enable --now earlyoom

For Arch Linux, there's an Arch Linux package:

sudo pacman -S earlyoom
sudo systemctl enable --now earlyoom

Availability in other distributions: see repology page.

Use

Just start the executable you have just compiled:

./earlyoom

It will inform you how much memory and swap you have, what the minimum is, how much memory is available and how much swap is free.

./earlyoom
eearlyoom v1.8
mem total: 23890 MiB, user mem total: 21701 MiB, swap total: 8191 MiB
sending SIGTERM when mem avail <= 10.00% and swap free <= 10.00%,
        SIGKILL when mem avail <=  5.00% and swap free <=  5.00%
mem avail: 20012 of 21701 MiB (92.22%), swap free: 5251 of 8191 MiB (64.11%)
mem avail: 20031 of 21721 MiB (92.22%), swap free: 5251 of 8191 MiB (64.11%)
mem avail: 20033 of 21723 MiB (92.22%), swap free: 5251 of 8191 MiB (64.11%)
[...]

If the values drop below the minimum, processes are killed until it is above the minimum again. Every action is logged to stderr. If you are running earlyoom as a systemd service, you can view the last 10 lines using

systemctl status earlyoom

Testing

In order to see earlyoom in action, create/simulate a memory leak and let earlyoom do what it does:

tail /dev/zero

Checking Logs

If you need any further actions after a process is killed by earlyoom (such as sending emails), you can parse the logs by:

sudo journalctl -u earlyoom | grep sending

Example output for above test command (tail /dev/zero) will look like:

Feb 20 10:59:34 debian earlyoom[10231]: sending SIGTERM to process 7378 uid 1000 "tail": oom_score 156, VmRSS 4962 MiB

For older versions of earlyoom, use:

sudo journalctl -u earlyoom | grep -iE "(sending|killing)"

Notifications

Since version 1.6, earlyoom can send notifications about killed processes via the system d-bus. Pass -n to enable them.

To actually see the notifications in your GUI session, you need to have systembus-notify running as your user.

Additionally, earlyoom can execute a script for each process killed, providing information about the process via the EARLYOOM_PID, EARLYOOM_UID and EARLYOOM_NAME environment variables. Pass -N /path/to/script to enable.

Warning: In case of dryrun mode, the script will be executed in rapid succession, ensure you have some sort of rate-limit implemented.

Preferred Processes

The command-line flag --prefer specifies processes to prefer killing; likewise, --avoid specifies processes to avoid killing. See https://github.com/rfjakob/earlyoom/blob/master/MANPAGE.md#--prefer-regex for details.

Configuration file

If you are running earlyoom as a system service (through systemd or init.d), you can adjust its configuration via the file provided in /etc/default/earlyoom. The file already contains some examples in the comments, which you can use to build your own set of configuration based on the supported command line options, for example:

EARLYOOM_ARGS="-m 5 -r 60 --avoid '(^|/)(init|Xorg|ssh)$' --prefer '(^|/)(java|chromium)$'"

After adjusting the file, simply restart the service to apply the changes. For example, for systemd:

systemctl restart earlyoom

Please note that this configuration file has no effect on earlyoom instances outside of systemd/init.d.

Command line options

earlyoom v1.8
Usage: ./earlyoom [OPTION]...

  -m PERCENT[,KILL_PERCENT] set available memory minimum to PERCENT of total
                            (default 10 %).
                            earlyoom sends SIGTERM once below PERCENT, then
                            SIGKILL once below KILL_PERCENT (default PERCENT/2).
  -s PERCENT[,KILL_PERCENT] set free swap minimum to PERCENT of total (default
                            10 %).
                            Note: both memory and swap must be below minimum for
                            earlyoom to act.
  -M SIZE[,KILL_SIZE]       set available memory minimum to SIZE KiB
  -S SIZE[,KILL_SIZE]       set free swap minimum to SIZE KiB
  -n                        enable d-bus notifications
  -N /PATH/TO/SCRIPT        call script after oom kill
  -g                        kill all processes within a process group
  -d, --debug               enable debugging messages
  -v                        print version information and exit
  -r INTERVAL               memory report interval in seconds (default 1), set
                            to 0 to disable completely
  -p                        set niceness of earlyoom to -20 and oom_score_adj to
                            -100
  --ignore-root-user        do not kill processes owned by root
  --sort-by-rss             find process with the largest rss (default oom_score)
  --prefer REGEX            prefer to kill processes matching REGEX
  --avoid REGEX             avoid killing processes matching REGEX
  --ignore REGEX            ignore processes matching REGEX
  --dryrun                  dry run (do not kill any processes)
  --syslog                  use syslog instead of std streams
  -h, --help                this help text

See the man page for details.

Contribute

Bug reports and pull requests are welcome via github. In particular, I am glad to accept

• Use case reports and feedback

Implementation Notes

• We don't use procps/libproc2 because procps_pids_select(), for some reason, always parses /proc/$pid/status. This is relatively expensive, and we don't need it.

Changelog

• v1.8.2, 2024-05-07

  • Fixes in earlyoom.service systemd unit file
    • Add process_mrelease to allowed syscalls (commit)
    • Fix IPAddressDeny syntax (commit)
    • Allow -p (commit)

• v1.8.1, 2024-04-17

  • Fix trivial test failures caused by message rewording (commit)

• v1.8, 2024-04-15

  • Introduce user mem total / meminfo_t.UserMemTotal and calculate MemAvailablePercent based on it (commit, more info in man page)
  • Use process_mrelease (#266)
  • Support NO_COLOR (https://no-color.org/)
  • Don't get confused by processes with a zombie main thread (commit)
  • Add --sort-by-rss, thanks @RanHuang! This will select a process to kill acc. to the largest RSS instead of largest oom_score.
  • The Gitlab CI testsuite now also runs on Amazon Linux 2 and Oracle Linux 7.

• v1.7, 2022-03-05

  • Add -N flag to run a script every time a process is killed (commit, man page section)
  • Add -g flag to kill whole process group (#247)
  • Remove -i flag (ignored for compatibility), it does not work properly on Linux kernels 5.9+ (#234)
  • Hardening: Drop ambient capabilities on startup (#234)

• v1.6.2, 2020-10-14

  • Double-check memory situation before killing victim (commit)
  • Never terminate ourselves (#205)
  • Dump buffer on /proc/meminfo conversion error (#214)

• 1.6.1, 2020-07-07

  • Clean up dbus-send zombie processes (#200)
  • Skip processes with oom_score_adj=-1000 (210)

• 1.6, 2020-04-11

  • Replace old notify-send GUI notification logic with dbus-send / systembus-notify (#183)
    • -n/-N now enables the new logic
    • You need to have systembus-notify running in your GUI session for notifications for work
  • Handle /proc mounted with hidepid gracefully (issue #184)

• v1.5, 2020-03-22

  • -p: set oom_score_adj to -100 instead of -1000 (#170)
  • Allow using both -M and -m, and -S and -s. The lower value (converted to percentages) will be used.
  • Set memory report interval in earlyoom.default to 1 hour instead of 1 minute (#177)

• v1.4, 2020-03-01

  • Make victim selection logic 50% faster by lazy-loading process attributes
  • Log the user id uid of killed processes in addition to pid and name
  • Color debug log in light grey
  • Code clean-up
    • Use block-local variables where possible
    • Introduce PATH_LEN to replace several hardcoded buffer lengths
  • Expand testsuite (make test)
  • Run cppcheck when available
  • Add unit-test benchmarks (make bench)
  • Drop root privileges in systemd unit file earlyoom.service

• v1.3.1, 2020-02-27

  • Fix spurious testsuite failure on systems with a lot of RAM (issue #156)

• v1.3, 2019-05-26

  • Wait for processes to actually exit when sending a signal
    • This fixes the problem that earlyoom sometimes kills more than one process when one would be enough (issue #121)
  • Be more liberal in what limits to accepts for SIGTERM and SIGKILL (issue #97)
    • Don't exit with a fatal error if SIGTERM limit < SIGKILL limit
    • Allow zero SIGKILL limit
  • Reformat startup output to make it clear that BOTH swap and mem must be <= limit
  • Add notify_all_users.py helper script
  • Add CODE_OF_CONDUCT.md (Contributor Covenant 1.4) (#102)
  • Fix possibly truncated UTF8 app names in log output (#110)

• v1.2, 2018-10-28

  • Implement adaptive sleep time (= adaptive poll rate) to lower CPU usage further (issue #61)
  • Remove option to use kernel oom-killer (-k, now ignored for compatibility) (issue #80)
  • Gracefully handle the case of swap being added or removed after earlyoom was started (issue 62, commit)
  • Implement staged kill: first SIGTERM, then SIGKILL, with configurable limits (issue #67)

• v1.1, 2018-07-07

  • Fix possible shell code injection through GUI notifications (commit)
  • On failure to kill any process, only sleep 1 second instead of 10 (issue #74)
  • Send the GUI notification after killing, not before (issue #73)
  • Accept --help in addition to -h
  • Fix wrong process name in log and in kill notification (commit 1, commit 2, issue #52, issue #65, issue #194)
  • Fix possible division by zero with -S (commit)

• v1.0, 2018-01-28

  • Add --prefer and --avoid options (@TomJohnZ)
  • Add support for GUI notifications, add options -n and -N

• v0.12: Add -M and -S options (@nailgun); add man page, parameterize Makefile (@yangfl)

• v0.11: Fix undefined behavior in get_entry_fatal (missing return, commit)

• v0.10: Allow to override Makefile's VERSION variable to make packaging easier, add -v command-line option

• v0.9: If oom_score of all processes is 0, use VmRss to find a victim

• v0.8: Use a guesstimate if the kernel does not provide MemAvailable

• v0.7: Select victim by oom_score instead of VmRSS, add options -i and -d

• v0.6: Add command-line options -m, -s, -k

• v0.5: Add swap support

• v0.4: Add SysV init script (thanks @joeytwiddle), use the new MemAvailable from /proc/meminfo (needs Linux 3.14+, commit)

• v0.2: Add systemd unit file

• v0.1: Initial release

earlyoom's Issues

Why -p option is not enabled by default?

subj

Add a warning before a process is killed

How about a feature that would warn me if the free memory gets close to the minimum? This way I could first try stopping the process myself.

Installation of earlyoom.default

The default config installs to /etc/default/earlyoom.default

The service file expects it here:

EnvironmentFile=-/etc/default/earlyoom

This should fix it up:

--- Makefile.orig	2017-11-29 19:01:21.537246323 -0500
+++ Makefile	2017-11-29 19:01:25.327274351 -0500
@@ -36,8 +36,8 @@
 	sed "s|:TARGET:|$(PREFIX)$(BINDIR)|g;s|:SYSCONFDIR:|$(SYSCONFDIR)|g" $< > $@
 
 install-default: earlyoom.default
-	install -d $(DESTDIR)$(SYSCONFDIR)/default/
-	install -m 644 $< $(DESTDIR)$(SYSCONFDIR)/default/
+	install -d $(DESTDIR)$(SYSCONFDIR)/default
+	cp $< $(DESTDIR)$(SYSCONFDIR)/default/earlyoom
 
 install-bin: earlyoom
 	install -d $(DESTDIR)$(PREFIX)$(BINDIR)/

Thanks!

Low memory is not OOM

Out of memory (OOM) is an often undesired state of computer operation where no additional memory can be allocated for use by programs or the operating system. Such a system will be unable to load any additional programs, and since many programs may load additional data into memory during execution, these will cease to function correctly. This usually occurs because all available memory, including disk swap space, has been allocated.

In normal operation, earlyoom should not occur at all OOM, since preventing OOM is the main task of earlyoom, not a reaction to OOM that has already occurred.

What about replace "Out of memory!" in earlyoom stdout with "Preventing OOM!"?

Sleep 10 causes earlyoom not prevents OOM

I ran while true; do tail /dev/zero; done, and eventually Xorg has been killed because earlyoom slept.

Would it be useful to use rate of swapin as a threshold?

Hello,

In addition to the absolute amount of memory swapped out, would it make sense to use the quantity of swap-in or the amount of time spent waiting on swap-in (if available) as a trigger for kill?

My use case are servers that have overcommit off, which means they are very conservative on how much memory can be committed, and extent of concurrency of multiple processes winds up penalized. When one considers simple Python, Go, or even C programs can allocate hundreds of megabytes of virtual memory that they never use (cat does this for me, in fact), this is a meaningful problem.

Thus, the amount of swap I may wish to allocate be somewhat large.

However, these servers have soft-real-time performance requirements. It's not desirable to actually let memory be swapped in. It would be better to start killing things if swap-in pressure gets beyond the residual.

Thoughts?

Wrong victim name

I ran tail /dev/zero, and tail killed, but earlyoom 1.0 displays wrong name

mem avail: 0 MiB (0 %), swap free: 2425 MiB (41 %)
mem avail: 0 MiB (0 %), swap free: 1631 MiB (27 %)
mem avail: 0 MiB (0 %), swap free: 830 MiB (14 %)
Out of memory! avail: 0 MiB < min: 587 MiB
Killing process 1526 sudo
mem avail: 5057 MiB (86 %), swap free: 5564 MiB (94 %)
mem avail: 5196 MiB (88 %), swap free: 5566 MiB (94 %)

Serious bug in get_entry_fatal

In 0.10, get_entry_fatal does not return val in the good case.

This code only works by accident with -O0.

Preferred processes

First, I wanted to thank you all for this great software. It's been quite helpful. In particular, Firefox usually eats too much memory and causes my computer to freeze. Earlyoom certainly helps with that.

However, when Firefox updated to v57 recently, it split out its tabs into sub-processes (similiar to Chrome). However, the main Firefox process often has the highest oom_score and is killed by earlyoom. For some reason, doing this orphans the sub-processes and causes a huge CPU spike---and usually a crash.

Everything works fine when a sub-processes is killed. So ideally, earlyoom would be able to selectively kill a tab's sub-process and not the entire Firefox process.

If you have any recommendations about how to accomplish this, I would really appreciate them. Otherwise, I found earlier comments by @joeytwiddle about implementing a "priority kill." So, I forked his code and modified it slightly. I've been testing it for a few days and it seems to do the job quite well. If you'd like---and if it's okay by @joeytwiddle ---I'd be happy to submit a PR.

Avoid killing my active Chrome tab

I am a tab junkie. (130 tabs currently open in Chrome, and sometimes I run Firefox alongside it.)

I have started noticing a recurring issue than when my memory is nearly full and I open a heavy webpage, after a few seconds the tab I am working in gets killed!

The behaviour I would prefer is that the new tab would be left running, even though it is using a lot of memory, and an older tab/process would be killed.

(Example "heavy" pages are Google Maps, Facebook, and Google Drive spreadsheets.)

So I would like a process to have its badness reduced when it first starts, but this would gently slide back to the default over time.

I wonder if the start time of a processes is available from se.exec in /proc/*/sched_start... (Edit: No, I found what I needed in .../stat.)

Admittedly my proposed solution will NOT help if I was to switch to an old tab/app and cause it to start consuming a lot of memory. For that situation, it might be more appropriate track which process the user has been recently interacting with. But I fear that might not be easy to detect. A simpler alternative would be to reduce badness of active processes, hence favouring the destruction of processes that have been idle for some time.

Not informative output if swap enabled

Out of memory! avail: 0 MiB < min: 587 MiB

One this line is not enaugh. Earlyoom not displays swap status info.

I want to see stdout like this:

mem avail: 0 MiB < min: 587 MiB
swap free: 500 MiB < min: 587 MiB
Preventing OOM: killing process...

Priority kill

Basically I always run out of memory in the same scenario: I open Dota, I open a few Chromium tabs, and eventually when Dota starts the game I run out of memory. I want Chromium to be killed, not Dota.

So for me either banning a process from being killed, or prioritizing a process to get killed would do the trick. It would be nice if there was a configuration for that.

Unexpected behavior when add swap after start earlyoom

sudo ./earlyoom
[sudo] пароль для user:
earlyoom (unknown version)
mem total: 5875 MiB, min: 587 MiB (10 %)
swap total: 0 MiB, min: 0 MiB (10 %)
mem avail: 4912 MiB (83 %), swap free: 0 MiB (0 %)
mem avail: 4910 MiB (83 %), swap free: 0 MiB (0 %)
mem avail: 4910 MiB (83 %), swap free: 0 MiB (0 %)
mem avail: 4910 MiB (83 %), swap free: 0 MiB (0 %)
mem avail: 4906 MiB (83 %), swap free: 0 MiB (0 %)
mem avail: 4898 MiB (83 %), swap free: 0 MiB (0 %)
mem avail: 4901 MiB (83 %), swap free: 0 MiB (0 %)
mem avail: 4902 MiB (83 %), swap free: 0 MiB (0 %)
mem avail: 4902 MiB (83 %), swap free: 0 MiB (0 %)
mem avail: 4905 MiB (83 %), swap free: 0 MiB (0 %)
mem avail: 4905 MiB (83 %), swap free: 0 MiB (0 %)
mem avail: 4905 MiB (83 %), swap free: 0 MiB (0 %)
mem avail: 4905 MiB (83 %), swap free: 0 MiB (0 %)
mem avail: 4905 MiB (83 %), swap free: 0 MiB (0 %)
mem avail: 4878 MiB (83 %), swap free: 5875 MiB (100 %)
mem avail: 4878 MiB (83 %), swap free: 5875 MiB (100 %)
mem avail: 4878 MiB (83 %), swap free: 5875 MiB (100 %)
mem avail: 4879 MiB (83 %), swap free: 5875 MiB (100 %)
mem avail: 4879 MiB (83 %), swap free: 5875 MiB (100 %)
...
mem avail: 0 MiB (0 %), swap free: 1034 MiB (17 %)
mem avail: 0 MiB (0 %), swap free: 927 MiB (15 %)
mem avail: 0 MiB (0 %), swap free: 824 MiB (14 %)
mem avail: 2 MiB (0 %), swap free: 698 MiB (11 %)
mem avail: 0 MiB (0 %), swap free: 604 MiB (10 %)
mem avail: 0 MiB (0 %), swap free: 501 MiB (8 %)
mem avail: 0 MiB (0 %), swap free: 396 MiB (6 %)
mem avail: 0 MiB (0 %), swap free: 273 MiB (4 %)
mem avail: 0 MiB (0 %), swap free: 177 MiB (3 %)
mem avail: 0 MiB (0 %), swap free: 84 MiB (1 %)
Out of memory! avail: 0 MiB < min: 587 MiB
Killing process 2456 python3
mem avail: 1606 MiB (27 %), swap free: 472 MiB (8 %)
mem avail: 5184 MiB (88 %), swap free: 5572 MiB (94 %)
mem avail: 5177 MiB (88 %), swap free: 5573 MiB (94 %)
mem avail: 5173 MiB (88 %), swap free: 5574 MiB (94 %)
mem avail: 5173 MiB (88 %), swap free: 5574 MiB (94 %)

I expect that earlyoom kill victim if avail = 10%, not 0 MiB.

About 0.6MB RSS

please, fix earlyoom.1

Use linux/sysinfo.h ?

Parsing /proc/meminfo 10 times a second, could you for an even lighter footprint benefit from using linux/sysinfo.h interface?

http://stackoverflow.com/questions/349889/how-do-you-determine-the-amount-of-linux-system-ram-in-c

del -k

IMO -k option is deprecated and should be deleted at all.

broken install

$ git clone https://github.com/rfjakob/earlyoom.git
Клонирование в «earlyoom»…
remote: Counting objects: 568, done.
remote: Compressing objects: 100% (85/85), done.
remote: Total 568 (delta 69), reused 101 (delta 43), pack-reused 439
Получение объектов: 100% (568/568), 136.47 KiB | 0 bytes/s, готово.
Определение изменений: 100% (326/326), готово.
$ cd earlyoom
$ make
cc -Wall -Wextra -DVERSION="v1.1-12-g647107d" -g -fstack-protector-all -std=gnu99 -o earlyoom meminfo.c kill.c main.c sanitize.c
pandoc MANPAGE.md -s -t man > earlyoom.1
/bin/sh: 1: pandoc: not found
Makefile:22: ошибка выполнения рецепта для цели «earlyoom.1»
make: *** [earlyoom.1] Error 127

Alias for -h

This is a really small one. When you do --help it should show the help page, and when there is an unrecognized option it should suggest -h.
Cheers <3

"make install" does not install man page

When installing earlyoom, the program's manual page (earlyoom.1) is not installed. The attached, updated Makefile adds targets for "install-man" and "uninstall-man" which can be called directly or by an uninstall process.
Makefile-earlyoom.txt

Default syntax

Hi,

I'm running Debian Stretch and my regex skills are poor.

I put the following in /etc/default/earlyoom
``EARLYOOM_ARGS="-m 15 -s 100 -r 5 --prefer '(firefox-esr|Web\sContent|gvfsd-mtp)'"

If I do ps ax | grep earlyoom, I see:

/usr/bin/earlyoom -m 15 -s 100 -r 5 --prefer (firefox-esr|WebsContent|gvfsd-mtp)

which looks like the \s I did to match a space is being misinterpeted.

I am trying to match any of the three piggies: firefox-esr, Web Content, gvfsd-mtp and to ignore the state of Swap and kill based on RAM usage.

Am I doing it correctly?

Thanks!
Mark

Add a hint on how to ignore swap usage

I used -s 100 to practically ignore the free swap memory. Maybe this is worth hinting at when using ./earlyoom -h or man.

Unexpected behavior: unknown version instead of 1.0

subj

Notify-send slowdown

...if & not in the end of the command.

Replace
"-i dialog-warning 'earlyoom' 'Killing process %d %s'", victim_pid, victim_name);
with
"-i dialog-warning 'earlyoom' 'Killing process %d %s' &", victim_pid, victim_name);
will fix the problem

Notify-send as root

earlyoom -N 'sudo -u YOUR_USER DISPLAY=:0 DBUS_SESSION_BUS_ADDRESS=unix:path=/run/user/YOUR_USER_ID/bus notify-send' is not needed

earlyoom -N 'sudo -u YOUR_USER DISPLAY=:0 notify-send'
is completely enough

How much memory does earlyoom use?

1.8 MiB (Debian 9 x86_64)
2.0 MiB (Fedora 28)
2.2 MiB (Debian 8 i386, https://i.imgur.com/MaRPWwE.png)
Not about 0.6 MiB.

Decrease CPU usage

There is no sense to check memory 10 times per second if there is more than one gigabyte of available memory. You can optimize the frequency of memory checks to decrease the frequency of memory checks and decrease CPU usage. Now the sleep period is 0.1. You can find the time until the next check as follows:

t1 = MemAvailable / 4000000
t2 = SwapFree / 1000000
sleep_until_next_mem_check = t1 + t2

I implemented a similar algorithm in nohang, which is OOM preventer written in Python. If nohang checked memory 10 times per second, then the CPU usage would be too large. Now when using this algorithm for calculating the period between memory checks, the CPU usage is significant only at a low memory level. If there is a lot of memory available, then nohang can use the processor even less than earlyoom. Demo https://youtu.be/8vjeolxw7Uo

I think it's possible to improve earlyoom in the same way. In most cases it will reduce the CPU usage by an order of magnitude.

/etc/earlyoom.conf instead of /etc/default/earlyoom

earlyoom.conf may be looking like follow:

# This is earlyoom config file

# comment
mem_min_sigkill = 6 %

# comment
swap_min_sigkill = 200 M

# for chromium, comment how it works
decrease_oom_score_adj = False

# comment, how it works
desktop_notifications = False

# comment, examples
prefer = 

# comment
avoid = ssh|Xorg

# comment
self_nice = 0

# comment
self_oom_score_adj = 0

# comment
debug = False

Advantages is it more convenient for users, IMHO.
Users may easy find all defaults and what means each parameter.

What is disadvantages?

Zram support

What about zram support and mem_used_total as a trigger?
Now earlyoom does not help when writing the data with high entropy in zram swap.

support logging to separate file

would be nice to be able to do this when running in daemon mode.

Options for users who want to use swap too

My use case differs slightly from yours. I have a swap partition and I want to use it.

I want earlyoom to trigger when both the swap and the memory are nearly full.

I have been using this adapted version of earlyoom for a couple of weeks now, and it has served me well: https://github.com/joeytwiddle/earlyoom/tree/wait_for_swap_to_fill

However it is not compatible with your use case!

I wonder if we should make the effort to combine both these requirements together, so that one package can serve both users with your requirements, and users with my requirements.

What about SIGTERM?

2018 will end soon, but the signal is still not supported.

Cannot compile due to usage of long int

$ make
cc -Wextra -DVERSION=\"v0.10-4-g8ea46ee\" -g -o earlyoom main.c meminfo.c kill.c
main.c: In function ‘main’:
main.c:131:11: warning: format ‘%d’ expects argument of type ‘int’, but argument 3 has type ‘long int’ [-Wformat=]
    printf("mem avail: %lu MiB (%d %%), swap free: %lu MiB (%d %%)\n",
           ^
kill.c: In function ‘get_process_stats’:
kill.c:86:12: warning: use of assignment suppression and length modifier together in gnu_scanf format [-Wformat=]
  fscanf(f, "%*lu %lu", &(p.vm_rss));
            ^

Ubuntu 16.04

Not killing processes

#78
IMO it was unexpected behavior.

What happens if start earlyoom -m 50?
Is it true that the swap is then not counted at all?

How to free up caches/buffers?

earlyoom starts killing processes when just 10% of memory "available".

$ free -h
              total        used        free      shared  buff/cache   available
Mem:            15G         12G        660M        883M        2,7G        2,0G
Swap:            0B          0B          0B

How to force system to free up more buffer/caches under certain conditions so that it frees more memory to become available?

Misestimated free RAM

Earlyoom killed my app because I don't have enough free RAM:

$ sudo journalctl -b | tail
...
Jul 04 19:52:50 xylix earlyoom[699]: Out of memory! avail: 3157 MiB < min: 3160 MiB
Jul 04 19:52:50 xylix earlyoom[699]: Killing process 20118 (VirtualBox)
...

But I have more than enough (~19 GB) in disposable buffers/cache.

$ free -h
             total       used       free     shared    buffers     cached
Mem:           30G        27G       3.6G        13G       228M        15G
-/+ buffers/cache:        11G        19G
Swap:           0B         0B         0B

[FR] - Get notification when killing a process

Hello,

I really like earlyoom, but sometimes it kills some of my processes without no notification. I have to wonder what happened. e.g. python program is just silently ended without no information about being killed or whatsoever.

It would be really awesome if we could have some event on which notification would be triggered, e.g. the most simple one like:

notify-send 'EarlyOOM' 'Process {$name}:{$PID} killed' --icon=dialog-information

right now it could be implemented as a workaround, when watching stdout of systemd unit. But some nicer solution would be great.

Sanitize process name for GUI notification helper

Is it not enough to drop out only ', ", backslash and backtick? Why do you drop out all Unicode?

Error: Could not find a process to kill

Got some process to eat the whole of RAM. earlyoom appears to have crashed.

If it's relevant, I don't have swap.

$ sudo systemctl status earlyoom
● earlyoom.service
   Loaded: loaded (/etc/systemd/system/earlyoom.service; enabled)
   Active: failed (Result: exit-code) since Mon 2016-02-08 00:05:04 CET; 3 days ago
  Process: 699 ExecStart=/usr/local/bin/earlyoom (code=exited, status=9)
 Main PID: 699 (code=exited, status=9)

Feb 08 00:05:04 zoob earlyoom[699]: earlyoom v0.7-10-gb4c6bc0
Feb 08 00:05:04 zoob earlyoom[699]: mem total: 31600 MiB, min: 3160 MiB (10 %)
Feb 08 00:05:04 zoob earlyoom[699]: swap total: 0 MiB, min: 0 MiB (10 %)
Feb 08 00:05:04 zoob earlyoom[699]: Error: Could not find a process to kill
Feb 08 00:05:04 zoob systemd[1]: earlyoom.service: main process exited, code=exited, status=9/n/a
Feb 08 00:05:04 zoob systemd[1]: Unit earlyoom.service entered failed state.

Not killing processes

I installed the package from AUR, enabled (and started) the service and manually started it with
sudo earlyoom -m 50, but it does not work.

I started a jupyter notebook and got the memory down to 2% when it stopped (because it finished what it was loading), but earlyoom never even touched it.

output.txt

What am I doing wrong?

Way to tell earlyoom not to kill some process (respect oom_score_adj)

Hello,

I have some processes which can consume a lot of memory, but I don't want them to be killed by earlyoom. And I would also like to be able to set some processes to be preferably killed. So it would be really great if earlyoom could be told to respect oom_score_adj. I know there is problem for you with Chrome setting oom_score_adj too high, but for me this is desired behavior (rather kill browser tab than some memory hungry - but important - process, computation, compilation, etc.). I know I can compile it with USE_KERNEL_OOM_KILLER, which would respect oom_score_adj, but I want to use userspace killing mechanism (I don't want to enable sysrq "f" trigger and I have also bad experience with kernel oom killer in general).

Ideally there should be some more fine grained setting, like list of processes (by name) for which earlyoom should ignore oom_score_adj (this would solve your problem with Chrome) and setting how much should be oom_score_adj taken into account when deciding what to kill. But for the start, even simple (preferably runtime) option to respect/don't respect oom_score_adj would be enough.

Reduce badness of certain processes using regexp

I found earlyoom was too often killing the main firefox process, or the main chrome process, when I would really rather it had killed just one or two of the chrome tabs I had open, instead of killing a whole browser session!

My approach to this has been to reduce the badness of processes matching a certain regexp. I compare the regexp against the process cmdline, rather than just the process name, so that I can be kind to the main process, but not to tabs or extensions. (I divide badness by 32.)

My fork based on your older code is here: https://github.com/joeytwiddle/earlyoom/tree/logging_usleep

It has been working for me, and I found it very useful (so that I don't lose my whole browser when I have 100 tabs open). I hope you also like the idea, and will consider merging it in! (I could create a PR.)

Here is a more up-to-date version which is merged with your oom_adj code (which is kind of overkill!): https://github.com/joeytwiddle/earlyoom

However now I am interested in reducing the badness of recently-created or running processes. (Because I would prefer if earlyoom attacks chrome tabs I haven't used for a long time, rather than tabs I am using right now.) (I suspect recently started processes are much easier to detect in Linux than recently used... ;) ) That's now in #8

Makefile getting version only works when checking out from git repo.

I was hoping someone could fix this line in the make file such that it works if you download a release: VERSION ?= $(shell git describe --tags --dirty)

The following shell code might be a good starting point, but since I don't know make I'm not sure if it is appropriate.
git describe --tags --dirty 2> /dev/null || basename $PWD | sed 's/earlyoom-//'

This assumes you have downloaded a release like so

VERSION=0.11
wget -q https://github.com/rfjakob/earlyoom/archive/v${VERSION}.tar.gz
tar -xzf v${VERSION}.tar.gz
cd earlyoom-${VERSION}

As you can see the version is included in the filename so I propose using the file name as the basis for this. Though, I could see having a VERSION file would be much simpler.

Feel free to ignore me and just delete this issue (serious).

What is a good default memory/swap limit?

At various times I have tried 5%, 10% and 15%. What have you guys found works for you?

I found on both my machines, I need to set the OOM killer to engage when I have 300 MB RAM and 300 MB swap remaining. This is the point when the swap starts thrashing.

(Although curiously on my machine with 8 GB RAM and 2 GB swap, if I don't run earlyoom, eventually I will end up with 600 MB RAM and 0 MB swap remaining, and at this point the system becomes unresponsive.)

I wonder if a fixed value of 300 MB RAM and 300 MB swap would satisfy most users. (I don't see why someone with 16 GB RAM would need to engage the OOM killer earlier than someone with 8 GB RAM.)

But obviously on a small machine (e.g. with only 1Gig RAM), the 300MB threshold would be far too high! So perhaps we should be using a percentage, with 300 as the upper limit...?

Interested to hear what values others are using.

earlyoom crash: Could not convert number: Numerical result out of range

I just noticed earlyoom did this, next time I'll try to provide more debug info:

Out of memory! avail: 138 MiB < min: 786 MiB
Killing process 4084 kworker/u17:3
Out of memory! avail: 155 MiB < min: 786 MiB
Killing process 4084 kworker/u17:3
Out of memory! avail: 157 MiB < min: 786 MiB
Killing process 4084 kworker/u17:3
Could not convert number: Numerical result out of range

But I think earlyoom should never kill kernel threads ;) and not quit on that conversion error.

Can compile but running fails with: Could not find "MemAvailable:"

The new version compiles fine, but when I run it I get:

$ ./earlyoom
earlyoom v0.7-1-g131aa9c
Could not find "MemAvailable:"

Any idea how I can fix this?

(gcc 4.6.3, libc 2.15 on Ubuntu 12.04)

Sleep(10)

Closed too early. The discussion should be continued.

Firstly, sleep(10) not declared and users are not warned.

Secondly, sleep(10) is too long, sleep(3) is enough.

Set when to kill OOM when using `systemctl` (systemd)

Hey, just installed OOM, and I started earlyoom by doing sudo systemctl enable earlyoom and sudo systemctl start earlyoom. Where do I set the -m option for earlyoom?

Inside earlyoom.service:

ExecStart=/usr/local/bin/earlyoom $EARLYOOM_ARGS

Not sure where $EARLYOOM_ARGS is

Notification shows wrong process name

On Ubuntu 16.04, running as user (not as service) the notify-osd notification reported the correct PID but the wrong process name "gnome-system-monitor" when in fact killing whatsapp. The terminal output shows the correct PID and name.

Configuration for service

Is there possibility to set custom percentages and run as service?

Never-Kill-RegEx

Last suggestion for today:

While testing the script (setting the minimum RAM to ca. 95%) I noticed that it would even kill my desktop environment in order to free up some space. I guess there are more "realistic" cases where an option Do not kill those processes, whatever happens, would be useful, though. So how about adding this? :)