Identify it is OOMKill<\/p>\n
Reason should be OOMKill and the time is Finished<\/p>\n
kubectl get pods testapp-v092-p8czf -o yaml | less -i<\/p>\n
…
\n Last State: Terminated
\n Reason: OOMKilled
\n Exit Code: 137
\n Started: Fri, 11 Sep 2020 11:00:08 +0800
\n Finished: Mon, 14 Sep 2020 13:00:46 +0800
\n…<\/p>\n
OOM heap dump ( when oomkill happen )<\/p>\n
Container entrypoints add java start params<\/p>\n
-XX:+HeapDumpOnOutOfMemoryError -XX:HeapDumpPath= 2. Mount an emptyDir for the pod In the pod lifecycle, \/var\/log\/dump is durable.<\/p>\n 3. Compress the dumpfile and download<\/p>\n gzip heapdump2020-09-15-03-198523874477783269974.hprof Check List ( pod is already restarted )<\/p>\n check stackdirver applictaion logs<\/p>\n check memory and cpu limits<\/p>\n $ kubectl get pods testapp-v203-trsfl -o yaml check kubectl top status<\/p>\n $ kubectl top pod testapp-v203-trsfl –containers new relic pod memory:<\/p>\n commands investigate java stack heap (inside pod)<\/p>\n apk add –no-cache jattach –repository http:\/\/dl-cdn.alpinelinux.org\/alpine\/edge\/community\/ ps find RSS of process (inside pod)<\/p>\n $ kubectl exec -it testapp-v203-trsfl \/bin\/bash Issues:<\/p>\n","protected":false},"excerpt":{"rendered":" Identify it is OOMKill Reason should be OOMKill and the time is Finished kubectl get pods testapp-v092-p8czf -o yaml | less -i … Last State: Terminated Reason: OOMKilled Exit Code: 137 Started: Fri, 11 Sep 2020 11:00:08 +0800 Finished: Mon, 14 Sep 2020 13:00:46 +0800 … OOM heap dump ( when oomkill happen ) Container […]<\/p>\n","protected":false},"author":3,"featured_media":0,"comment_status":"open","ping_status":"open","sticky":false,"template":"","format":"standard","meta":{"footnotes":""},"categories":[7],"tags":[],"class_list":["post-9107","post","type-post","status-publish","format-standard","hentry","category-life"],"_links":{"self":[{"href":"https:\/\/jpuyy.com\/index.php?rest_route=\/wp\/v2\/posts\/9107","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/jpuyy.com\/index.php?rest_route=\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/jpuyy.com\/index.php?rest_route=\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/jpuyy.com\/index.php?rest_route=\/wp\/v2\/users\/3"}],"replies":[{"embeddable":true,"href":"https:\/\/jpuyy.com\/index.php?rest_route=%2Fwp%2Fv2%2Fcomments&post=9107"}],"version-history":[{"count":8,"href":"https:\/\/jpuyy.com\/index.php?rest_route=\/wp\/v2\/posts\/9107\/revisions"}],"predecessor-version":[{"id":9235,"href":"https:\/\/jpuyy.com\/index.php?rest_route=\/wp\/v2\/posts\/9107\/revisions\/9235"}],"wp:attachment":[{"href":"https:\/\/jpuyy.com\/index.php?rest_route=%2Fwp%2Fv2%2Fmedia&parent=9107"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/jpuyy.com\/index.php?rest_route=%2Fwp%2Fv2%2Fcategories&post=9107"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/jpuyy.com\/index.php?rest_route=%2Fwp%2Fv2%2Ftags&post=9107"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}
\nkubectl cp testapp-v1127-xnbhq:\/var\/log\/dump\/heapdump2020-09-15-03-198523874477783269974.hprof.gz \/tmp\/heapdump2020-09-15-03-198523874477783269974.hprof.gz<\/p>\n
\n …
\n resources:
\n limits:
\n cpu: 1500m
\n memory: 1229Mi
\n requests:
\n cpu: 300m
\n memory: 1Gi
\n …<\/p>\n
\nPOD NAME CPU(cores) MEMORY(bytes)
\ntestapp-v203-trsfl testapp 13m 1144Mi
\ntestapp-v203-trsfl istio-proxy 5m 47Mi<\/p>\n
\njattach pid inspectheap
\njattach pid jcmd VM.info<\/p>\n
\nps -o pid,user,vsz,rss,comm,args
\nPID USER VSZ RSS COMMAND COMMAND
\n 1 root 4332 720 tini \/tini — \/entrypoint.sh java
\n 7 test 6.3g 1.1g java java -XX:+UseG1GC -Xms768m -Xmx768m -DREGION=gcp_hk -XX:+ExitOnOutOfMemoryError -XX:+UseStringDeduplication -XX:StringDeduplicationAgeThreshold=3 -agentlib:jdwp=transport=dt_socket,ser
\n18215 root 2620 2316 bash \/bin\/bash
\n18267 root 1572 20 ps ps -o pid,user,vsz,rss,comm,args<\/p>\n