smpio / kube-oom-monitor

Kubernetes OOM event monitor

kube-oom-monitor

kube-oom-monitor continuously monitors kernel OOM messages and creates Kubernetes Events with PID and cgroup. This information can be used to determine in which container of which pod OOM occurred. It is supposed to be runned as DaemonSet.

Requiremenets

Container should be run in priveleged context:

securityContext:
  privileged: true

Usage

-nodeName string
    name of the node to bind events (required)
-eventReason string
    event reason (default "NodeOOM")

Background

Kubernetes node-problem-detector continuously reads /dev/kmsg, parses OOM messages and creates Kubernetes event OOMKilling. But it does not store cgroup information of killed process. Without this information it's hard to guess container and pod it relates to.

There is also SystemOOM event sometimes created by kubelet. But it doesn't pass cgroup too, although it uses cadvisor oomparser, that provides it!

kube-oom-monitor uses the same cadvisor oomparser.

kube-oom-monitor's Issues

Don't create events for old OOMs

Incorrect `firstTimestamp` and `lastTimestamp`

firstTimestamp = lastTimestamp ~= actualTime - 6 minutes