Applies To: CENTOS DEBIAN UBUNTU

What is the OOM Killer?

The "OOM Killer" or "Out of Memory Killer" is a process that the Linux kernel employs when the system is critically low on memory. This situation occurs because processes on the server are consuming a large amount of memory, and the system requires more memory for its own processes and to allocate to other processes. When a process starts it requests a block of memory from the kernel. This initial request is usually a large request that the process will not immediately or indeed ever use all of. The kernel, aware of this tendency for processes to request redundant memory, over allocates the system memory. This means that when the system has, for example, 8GB of RAM the kernel may allocate 8.5GB to processes. This maximises the use of system memory by ensuring that the memory that is allocated to processes is being actively used.

Normally, this situation does not cause a problem. However, if enough processes begin to use all of their requested memory blocks then there will not be enough physical memory to support them all. This means that the running processes require more memory than is physically available. This situation is critical and must be resolved immediately.

The solution that the Linux kernel employs is to invoke the OOM Killer to review all running processes and kill one or more of them in order to free up system memory and keep the system running.

How does it select a Process to Kill?

The OOM Killer works by reviewing all running processes and assigning them a badness score. The process that has the highest score is the one that is killed. The OOM Killer assigns a badness score based on a number of criteria. The principle of which are as follows are as follows:

  • The process and its all of its child processes are using a lot of memory.

  • The minimum number of processes are killed (ideally one) in order to free up enough memory to resolve the situation.

  • Root, kernel and important system processes are given much lower scores.

So why is Apache / MySQL always Killed?

The above listed criteria mean that when selecting a process to kill the OOM Killer will choose a process using lots of memory and has lots of child processes and which are not system processes. An application such as Apache, MySQL, Nginx, Clamd (ClamAV), or a mail server will make an good candidate. However, as this situation usually occurs on a busy web servers Apache or MySQL will be the largest in-memory, non-system processes and consequently gets killed.

It must be remembered that although the Web Server or DB Server are very important to you, when the kernel calls the OOM Killer the situation is critical. If memory is not freed by killing a process the server will crash very shortly afterwards. Continuing normal operations at this juncture is impossible.

Common Cause

One of the common causes of Apache / Nginx / MySQL being killed by the OOM Process Killer, is when the site is receiving a large amount of traffic, this could be genuine traffic from a New Promotion, Media Attention or similar, or it could be a Bot crawling the site, or in some cases it can be Botnets, trying to attack of brute force your site. Reviewing the Apache / Nginx logs, is a good place to start to see if this is the case.


How to find if the OOM Process Killer was recently nvoked?

The easiest way to find if the OOM Killer was invoked and potentially the reason that a website went offline or similar, is to check the system logs. Whenever the OOM Killer is invoked it will write a great deal of information to the system log including which process was killed and why. You can run the following commands;

dmesg | egrep -i “killed process”
BASH

Additionally, the below can be used to look through the OS Distribution Specific Logs;

#CentOS
grep -i "out of memory" /var/log/messages

#Debian / Ubuntu
grep -i "out of memory" /var/log/kern.log
BASH

The output will look like something like this,

host kernel: Out of Memory: Killed process 2592 (mysql).
TEXT

In this instance the process that was killed was mysql which has the PID (Process Identification Number) of 2592.

Grep Commands

When using the Grep Commands above, you can always add additional options to show more context, the line numbers, or to highlight the output.


How to avoid the OOM Process Killer

The OOM Killer will only get invoked when the system is critically low on memory. Consequently the solution to avoiding it is to either reduce the memory requirements of the server or increase the available memory.

Upgrade the Server

This is the fastest and easiest option as it will require no work re-configuring the server. If you would like to upgrade your server you can contact your Account Manager or our Sales Team.

Reduce Memory Usage

The first step that should be taken to reduce memory usage to stop any processes running that are not needed. For example if the server is not shared and FTP is only occasionally used then this process can be initiated prior to uploading and terminated afterwards.

WHM / cPanel servers generally have a lot of processes running which consume a lot of memory. Some of these can be stopped if they are not being used. This is done via the “Service Manager” page in WHM. The documentation for this feature can be read here.

The second way to reduce memory usage is use an alternative application or configuration that has a lower memory footprint. An obvious candidate for this is Apache. Apache is a very capable web server but is not know for is parsimonious use of system resources. There are very many alternative web servers but two well know web servers which show a decreased memory usage compared to apache are nginx and lighttpd.

Swapping to nginx from apache will make a large difference to memory usage but only on a busy web server.

Alternatively, using a reverse proxy to serve static content will reduce memory usage and increase page load times by serving content directly from RAM before apache has to start a thread to serve it. Varnish cache is an excellent reverse proxy. Further information can be found on their site:

https://www.varnish-cache.org/

WHM / cPanel servers do not have very much choice in which application to use as often the only available option is the one supplied by WHM. 

Choosing alternative applications or tweaking configuration will only produce limited results. Increasing the available RAM to an amount sufficient to support the needed processes is always the best solution.