Javascript heap out of memory while running a js script to fetch data from an api every minute- javascript/node.js

user_15540554 0
Viewed 12349

My program grabs ~70 pages of 1000 items from an API and bulk-inserts it into a SQLite database using Sequelize. After looping through a few times, the memory usage of node goes up to around 1.2GB and and then eventually crashes the program with this error: FATAL ERROR: MarkCompactCollector: young object promotion failed Allocation failed - JavaScript heap out of memory. I've tried using delete for all of the big variables that I use for the response of the API call and stuff with variable = undefined and then global.gc(), however I still get huge amounts of memory usage and eventually it crashes. Would increasing the memory cap of Node.js help? Or would the memory usage of it just keep increasing until it hits the next cap?

Here's the full output of the error:


<--- Last few GCs --->

[6760:0x128008000]   436085 ms: Scavenge 4068.7 (4110.5) -> 4068.7 (4110.5) MB, 2.7 / 0.0 ms  (average mu = 0.918, current mu = 0.875) allocation failure 
[6760:0x128008000]   436345 ms: Scavenge 4073.0 (4113.8) -> 4072.9 (4118.8) MB, 9.2 / 0.0 ms  (average mu = 0.918, current mu = 0.875) allocation failure 
[6760:0x128008000]   436565 ms: Scavenge (reduce) 4079.1 (4122.1) -> 4079.3 (4121.9) MB, 4.6 / 0.0 ms  (average mu = 0.918, current mu = 0.875) allocation failure 


<--- JS stacktrace --->


<--- Last few GCs --->

[6760:0x128008000]   436085 ms: Scavenge 4068.7 (4110.5) -> 4068.7 (4110.5) MB, 2.7 / 0.0 ms  (average mu = 0.918, current mu = 0.875) allocation failure 
[6760:0x128008000]   436345 ms: Scavenge 4073.0 (4113.8) -> 4072.9 (4118.8) MB, 9.2 / 0.0 ms  (average mu = 0.918, current mu = 0.875) allocation failure 
[6760:0x128008000]   436565 ms: Scavenge (reduce) 4079.1 (4122.1) -> 4079.3 (4121.9) MB, 4.6 / 0.0 ms  (average mu = 0.918, current mu = 0.875) allocation failure 


<--- JS stacktrace --->

FATAL ERROR: MarkCompactCollector: young object promotion failed Allocation failed - JavaScript heap out of memory
FATAL ERROR: MarkCompactCollector: young object promotion failed Allocation failed - JavaScript heap out of memory

<--- Last few GCs --->

[6760:0x128008000]   436085 ms: Scavenge 4068.7 (4110.5) -> 4068.7 (4110.5) MB, 2.7 / 0.0 ms  (average mu = 0.918, current mu = 0.875) allocation failure 
[6760:0x128008000]   436345 ms: Scavenge 4073.0 (4113.8) -> 4072.9 (4118.8) MB, 9.2 / 0.0 ms  (average mu = 0.918, current mu = 0.875) allocation failure 
[6760:0x128008000]   436565 ms: Scavenge (reduce) 4079.1 (4122.1) -> 4079.3 (4121.9) MB, 4.6 / 0.0 ms  (average mu = 0.918, current mu = 0.875) allocation failure 


<--- JS stacktrace --->

FATAL ERROR: MarkCompactCollector: young object promotion failed Allocation failed - JavaScript heap out of memory

<--- Last few GCs --->

[6760:0x128008000]   436085 ms: Scavenge 4068.7 (4110.5) -> 4068.7 (4110.5) MB, 2.7 / 0.0 ms  (average mu = 0.918, current mu = 0.875) allocation failure 
[6760:0x128008000]   436345 ms: Scavenge 4073.0 (4113.8) -> 4072.9 (4118.8) MB, 9.2 / 0.0 ms  (average mu = 0.918, current mu = 0.875) allocation failure 
[6760:0x128008000]   436565 ms: Scavenge (reduce) 4079.1 (4122.1) -> 4079.3 (4121.9) MB, 4.6 / 0.0 ms  (average mu = 0.918, current mu = 0.875) allocation failure 


<--- JS stacktrace --->

FATAL ERROR: MarkCompactCollector: young object promotion failed Allocation failed - JavaScript heap out of memory
zsh: segmentation fault  npm start

I'm running node v15.12.0 on a MacBook Air M1 with 16GB of ram, however I don't think its the hardware that's the issue here as far as I understand this... Does anyone know why this is happening? Thanks in advance :)

Edit:

Turns out one of the node modules I was using never removed it's responses from the api calls, so I just rewrote that section of my code and now I'm good to go.

sequelize.js v8 node.js sqlite javascript
1 Answers

From the data you've provided, it's impossible to tell why you're running out of memory.

Maybe the working set (i.e. the amount of stuff that you need to keep around at the same time) just happens to be larger than your current heap limit; in that case increasing the limit would help. It's easy to find out by trying it, e.g. with --max-old-space-size=8000 (megabytes).

Maybe there's a memory leak somewhere, either in your own code, or in one of your third-party modules. In other words, maybe you're accidentally keeping objects reachable that you don't really need any more.

If you provide a repro case, then people can investigate and tell you more.

Side notes:

  • according to your output, heap memory consumption is growing to ~4 GB; not sure why you think it tops out at 1.2 GB.
  • it is never necessary to invoke global.gc() manually; the garbage collector will kick in automatically when memory pressure is high. That said, if something is keeping old objects reachable, then the garbage collector can't do anything.
user_60364280
Related