k8s pod OOMKill Exit Code: 137

Identify it is OOMKill

Reason should be OOMKill and the time is Finished

kubectl get pods testapp-v092-p8czf -o yaml | less -i

…
Last State: Terminated
Reason: OOMKilled
Exit Code: 137
Started: Fri, 11 Sep 2020 11:00:08 +0800
Finished: Mon, 14 Sep 2020 13:00:46 +0800
…

OOM heap dump ( when oomkill happen )

Container entrypoints add java start params

-XX:+HeapDumpOnOutOfMemoryError -XX:HeapDumpPath=\.hprof

2. Mount an emptyDir for the pod In the pod lifecycle, /var/log/dump is durable.

3. Compress the dumpfile and download

gzip heapdump2020-09-15-03-198523874477783269974.hprof
kubectl cp testapp-v1127-xnbhq:/var/log/dump/heapdump2020-09-15-03-198523874477783269974.hprof.gz /tmp/heapdump2020-09-15-03-198523874477783269974.hprof.gz

Check List ( pod is already restarted )

check stackdirver applictaion logs

check memory and cpu limits

$ kubectl get pods testapp-v203-trsfl -o yaml
…
resources:
limits:
cpu: 1500m
memory: 1229Mi
requests:
cpu: 300m
memory: 1Gi
…

check kubectl top status

$ kubectl top pod testapp-v203-trsfl –containers
POD NAME CPU(cores) MEMORY(bytes)
testapp-v203-trsfl testapp 13m 1144Mi
testapp-v203-trsfl istio-proxy 5m 47Mi

new relic pod memory:

commands investigate java stack heap (inside pod)

apk add –no-cache jattach –repository http://dl-cdn.alpinelinux.org/alpine/edge/community/
jattach pid inspectheap
jattach pid jcmd VM.info

ps find RSS of process (inside pod)

$ kubectl exec -it testapp-v203-trsfl /bin/bash
ps -o pid,user,vsz,rss,comm,args
PID USER VSZ RSS COMMAND COMMAND
1 root 4332 720 tini /tini — /entrypoint.sh java
7 test 6.3g 1.1g java java -XX:+UseG1GC -Xms768m -Xmx768m -DREGION=gcp_hk -XX:+ExitOnOutOfMemoryError -XX:+UseStringDeduplication -XX:StringDeduplicationAgeThreshold=3 -agentlib:jdwp=transport=dt_socket,ser
18215 root 2620 2316 bash /bin/bash
18267 root 1572 20 ps ps -o pid,user,vsz,rss,comm,args

Issues: