What happens when a pod exceeds its memory limit

Question

When a pod exceeds its memory limit? What happens?

Answer 1

• Whenever a pod exceeds its memory limit a signal SIGKILL is sent which  immediately terminates the container and spawns a new one with OOM(out of memory) error. 

• The OS, if using a cgroup based containerisation (docker, rkt, etc), will do the OOM killing. 

• Kubernetes simply sets the cgroup limits but is not ultimately responsible for killing the processes.

• 'SIGTERM ' is sent to PID 1 and k8s waits for (default of 30 seconds) `terminationGracePeriodSeconds` before sending the `SIGKILL` or you can change that time with terminationGracePeriodSeconds in the pod.

• As long as your container will eventually exit, it should be fine to have a long grace period. If you want a graceful restart it would have to do it inside the pod. 

• If you don't want it killed, then you shouldn't set a memory `limit` on the pod and there's not a way to disable it for the whole node. 

• Also, when the liveness probe fails, the container will SIGTERM and SIGKILL after some grace period.

Answer 2

( Note that the memory limit is set at the container level and not at the pod level )

In most CRI Implementations, Kubernetes only sets the cgroups and it is the job of the OS (  OOM killer ) to kill the process if it exceeds the memory limit.

Also note that if the main process has forked and created multiple processes inside the container, ( Or if you are exec ing inside the container using kubectl ) , OS will not kill all processes within the group. Instead,  OS will decide  ( using oom score )  on a single process to kill first. If this process is not the PID 1 , then the container will not restart. ( So, if you use a third party program like stress-ng -  which forks multiple times - in order to stress test the application, make sure you pay attention to teh logs )