Skip to content

OOM Basics

When a Linux system exhausts available memory, the kernel employs the Out-Of-Memory (OOM) killer to terminate processes and reclaim memory. This mechanism ensures the system remains functional by prioritizing critical processes. Understanding how the OOM killer operates is essential for diagnosing memory-related crashes and optimizing resource management.

OOM Basics and the OOM Killer

The OOM killer is a kernel component that identifies and terminates processes when memory allocation fails. It is invoked by the OOM reaper, a separate process that handles memory allocation errors. When a process cannot allocate memory, the OOM reaper triggers the OOM killer to free up resources by killing the most "offensive" process.

The OOM killer does not act arbitrarily. It uses a score-based algorithm to select the process to terminate, considering factors like memory usage, runtime, and process priority.

OOM Killer Mechanics

  1. Score Calculation:
    The OOM killer assigns a score to each process based on:
  2. Memory usage: Processes using more memory receive higher scores.
  3. Runtime: Longer-running processes are prioritized to avoid abrupt termination of short-lived tasks.

  4. OOM score adj: A tunable parameter (/proc/<pid>/oom_score_adj) that adjusts the score. Lower values (e.g., -1000) reduce the likelihood of termination, while higher values (e.g., +1000) increase it.

  5. OOM Score Adj:
    The oom_score_adj value ranges from -1000 to +1000. By default, it is 0. You can manually adjust it to influence the OOM killer's decision: 

    # Check current oom_score_adj for a process
cat /proc/<pid>/oom_score_adj

# Temporarily adjust the score (e.g., reduce likelihood of termination)
echo -1000 | sudo tee /proc/<pid>/oom_score_adj

  6. OOM Killer Logs:
    When the OOM killer terminates a process, it logs messages to the kernel ring buffer. Use dmesg or check /var/log/messages (systemd-based systems) to identify the victim: 

    Out of memory: Kill process <pid> (process_name) score <score> or sacrifice child

OOM Score Calculation Formula

The OOM killer calculates the score using a formula that weights memory usage and runtime. For example: - A process using 100MB of memory and running for 10 seconds might have a score of 100 * 10 = 1000. - The OOM killer then kills the process with the highest score.

Note: The exact algorithm varies by kernel version and may include additional factors like memory type (e.g., anonymous vs. file-backed pages).

Practical Examples

  1. Adjusting OOM Scores:

    # Prevent a critical service (e.g., nginx) from being killed
echo -500 | sudo tee /proc/$(pgrep nginx)/oom_score_adj

  2. Monitoring OOM Events:

    # View OOM killer logs
dmesg | grep -i 'out of memory'

  3. Analyzing Memory Usage:

    # Check memory usage and OOM score for all processes
ps -e -o pid,comm,pmem,pcpu,oom_score_adj

Key takeaways

  • The OOM killer terminates processes when memory is critically low, using a score-based algorithm.
  • oom_score_adj allows manual tuning of a process's priority in OOM scenarios.
  • OOM killer logs provide critical insights into which processes were terminated.
  • Understanding memory scoring helps optimize system stability and avoid unintended process termination.