I have ~650K documents I am adding to a mongo collection. I cannot tell why this is such a pain.
I am creating a js file that contains the documents and using mongosh.
One. I can add the documents with a insertOne and it is totally reliable and would take many, many, many hours.
Two, I can create a 1000 line insertMany call and I send it to mongosh and it works.
But. If I take the rest of the documents, put them into 1000 line buckets and send these to mongosh, it crashes. See below.
I could chop the lines into chunks smaller than 1000, such as 500 or 200. I can chop them into chunks, put them into separate files and create a script to call mongosh with each file. Or I am sure there are other hoops I can set up and jump through. But what the heck? Why am I having to do this? Life is too short.
I have a lot of RAM on this machine. Why is it so difficult to get apps to use the RAM I put into the machine for them to use? Urg.
the crash:
$ time mongosh "mongodb+srv://HOST/jdi-stats" --username api-reader --password ZEKRET --file 04.jsa
Current Mongosh Log ID: 632d399bf13a696d49897a8b
Connecting to: mongodb+srv://<credentials>@HOST/jdi-stats?appName=mongosh+1.5.4
Using MongoDB: 4.4.15
Using Mongosh: 1.5.4
For mongosh info see: https://docs.mongodb.com/mongodb-shell/
Loading file: 04.jsa
<--- Last few GCs --->
[117290:0xa6c8ef0] 27356 ms: Scavenge 4031.3 (4107.0) -> 4030.0 (4127.7) MB, 8.0 / 0.0 ms (average mu = 0.271, current mu = 0.221) allocation failure
[117290:0xa6c8ef0] 27383 ms: Scavenge 4043.6 (4127.7) -> 4039.9 (4129.2) MB, 16.4 / 0.0 ms (average mu = 0.271, current mu = 0.221) allocation failure
[117290:0xa6c8ef0] 28794 ms: Mark-sweep 4045.6 (4129.2) -> 4042.5 (4143.5) MB, 1405.5 / 0.0 ms (average mu = 0.191, current mu = 0.065) allocation failure scavenge might not succeed
<--- JS stacktrace --->
FATAL ERROR: Ineffective mark-compacts near heap limit Allocation failed - JavaScript heap out of memory
1: 0xb7ca10 node::Abort() [mongosh mongodb+srv://<credentials>@HOST/jdi-stats]
2: 0xa86201 node::FatalError(char const*, char const*) [mongosh mongodb+srv://<credentials>@HOST/jdi-stats]
3: 0xd6bede v8::Utils::ReportOOMFailure(v8::internal::Isolate*, char const*, bool) [mongosh mongodb+srv://<credentials>@HOST/jdi-stats]
4: 0xd6c257 v8::internal::V8::FatalProcessOutOfMemory(v8::internal::Isolate*, char const*, bool) [mongosh mongodb+srv://<credentials>@HOST/jdi-stats]
5: 0xf23605 [mongosh mongodb+srv://<credentials>@HOST/jdi-stats]
6: 0xf240e6 [mongosh mongodb+srv://<credentials>@HOST/jdi-stats]
7: 0xf3260e [mongosh mongodb+srv://<credentials>@HOST/jdi-stats]
8: 0xf33050 v8::internal::Heap::CollectGarbage(v8::internal::AllocationSpace, v8::internal::GarbageCollectionReason, v8::GCCallbackFlags) [mongosh mongodb+srv://<credentials>@HOST/jdi-stats]
9: 0xf35fae v8::internal::Heap::AllocateRawWithRetryOrFailSlowPath(int, v8::internal::AllocationType, v8::internal::AllocationOrigin, v8::internal::AllocationAlignment) [mongosh mongodb+srv://<credentials>@HOST/jdi-stats]
10: 0xef777a v8::internal::Factory::NewFillerObject(int, bool, v8::internal::AllocationType, v8::internal::AllocationOrigin) [mongosh mongodb+srv://<credentials>@HOST/jdi-stats]
11: 0x1273006 v8::internal::Runtime_AllocateInYoungGeneration(int, unsigned long*, v8::internal::Isolate*) [mongosh mongodb+srv://<credentials>@HOST/jdi-stats]
12: 0x165b5d9 [mongosh mongodb+srv://<credentials>@HOST/jdi-stats]
Aborted (core dumped)
real 1m46.195s
user 0m55.926s
sys 0m3.521s