Out of memory - Google Kubernetes

user_3364179 0
Viewed 393

I have an autopilot cluster which should increase the cpu/memory on demand but I am still getting the following error

I have not defined any limits/resources in the deployment. I am relying on Google to handle that automatically.

It's a node application that reads large csv files (300-400mb), parses them and inserts them to mysql db (using typeorm) It seems it works with smaller files. The files are read one by one. In this case there are over 1200 files (not all 300-400mb of size but quite many are)

It seems it does not work like I thought it would.... Is this a sign that there is something wrong with the js code or do I just need to increase the memory manually ?

<--- Last few GCs --->
[1:0x7f9fde9bf330]  4338991 ms: Scavenge (reduce) 955.5 (1037.2) -> 955.5 (1037.2) MB, 7.1 / 0.0 ms  (average mu = 0.295, current mu = 0.279) allocation failure
[1:0x7f9fde9bf330]  4339001 ms: Scavenge (reduce) 956.9 (1037.7) -> 956.9 (1038.0) MB, 7.6 / 0.0 ms  (average mu = 0.295, current mu = 0.279) allocation failure
[1:0x7f9fde9bf330]  4339011 ms: Scavenge (reduce) 958.1 (1038.0) -> 958.1 (1038.5) MB, 7.7 / 0.0 ms  (average mu = 0.295, current mu = 0.279) allocation failure
<--- JS stacktrace --->
FATAL ERROR: Ineffective mark-compacts near heap limit Allocation failed - JavaScript heap out of memory
google-kubernetes-engine node.js out-of-memory
1 Answers

I have not defined any limits/resources in the deployment. I am relying on Google to handle that automatically.

I don't think that's true. Copy pasting from the docs: "Autopilot relies on what you specify in your deployment configuration to provision resources. If you do not specify resource requests for any container in the Pod, Autopilot applies default values. These defaults are designed to give the containers in your Pods an average amount of resources, which are suitable for many smaller workloads."

And this: "Important: Google recommends that you explicitly set your resource requests for each container to meet your application requirements, as these default values might not be sufficient, or optimal."

So it's likely that the default resource requests and limit are too low for your application. You should set them to be a value high enough so you don't run out of memory.

user_3764450
Related