how can I add many small mongo documents without crashing?

Viewed 15

I have ~650K documents I am adding to a mongo collection. I cannot tell why this is such a pain.

I am creating a js file that contains the documents and using mongosh.

One. I can add the documents with a insertOne and it is totally reliable and would take many, many, many hours.

Two, I can create a 1000 line insertMany call and I send it to mongosh and it works.

But. If I take the rest of the documents, put them into 1000 line buckets and send these to mongosh, it crashes. See below.

I could chop the lines into chunks smaller than 1000, such as 500 or 200. I can chop them into chunks, put them into separate files and create a script to call mongosh with each file. Or I am sure there are other hoops I can set up and jump through. But what the heck? Why am I having to do this? Life is too short.

I have a lot of RAM on this machine. Why is it so difficult to get apps to use the RAM I put into the machine for them to use? Urg.

the crash:

$ time mongosh "mongodb+srv://HOST/jdi-stats" --username api-reader --password ZEKRET --file 04.jsa
Current Mongosh Log ID: 632d399bf13a696d49897a8b
Connecting to:      mongodb+srv://<credentials>@HOST/jdi-stats?appName=mongosh+1.5.4
Using MongoDB:      4.4.15
Using Mongosh:      1.5.4

For mongosh info see: https://docs.mongodb.com/mongodb-shell/

Loading file: 04.jsa

<--- Last few GCs --->

[117290:0xa6c8ef0]    27356 ms: Scavenge 4031.3 (4107.0) -> 4030.0 (4127.7) MB, 8.0 / 0.0 ms  (average mu = 0.271, current mu = 0.221) allocation failure 
[117290:0xa6c8ef0]    27383 ms: Scavenge 4043.6 (4127.7) -> 4039.9 (4129.2) MB, 16.4 / 0.0 ms  (average mu = 0.271, current mu = 0.221) allocation failure 
[117290:0xa6c8ef0]    28794 ms: Mark-sweep 4045.6 (4129.2) -> 4042.5 (4143.5) MB, 1405.5 / 0.0 ms  (average mu = 0.191, current mu = 0.065) allocation failure scavenge might not succeed


<--- JS stacktrace --->

FATAL ERROR: Ineffective mark-compacts near heap limit Allocation failed - JavaScript heap out of memory
 1: 0xb7ca10 node::Abort() [mongosh mongodb+srv://<credentials>@HOST/jdi-stats]
 2: 0xa86201 node::FatalError(char const*, char const*) [mongosh mongodb+srv://<credentials>@HOST/jdi-stats]
 3: 0xd6bede v8::Utils::ReportOOMFailure(v8::internal::Isolate*, char const*, bool) [mongosh mongodb+srv://<credentials>@HOST/jdi-stats]
 4: 0xd6c257 v8::internal::V8::FatalProcessOutOfMemory(v8::internal::Isolate*, char const*, bool) [mongosh mongodb+srv://<credentials>@HOST/jdi-stats]
 5: 0xf23605  [mongosh mongodb+srv://<credentials>@HOST/jdi-stats]
 6: 0xf240e6  [mongosh mongodb+srv://<credentials>@HOST/jdi-stats]
 7: 0xf3260e  [mongosh mongodb+srv://<credentials>@HOST/jdi-stats]
 8: 0xf33050 v8::internal::Heap::CollectGarbage(v8::internal::AllocationSpace, v8::internal::GarbageCollectionReason, v8::GCCallbackFlags) [mongosh mongodb+srv://<credentials>@HOST/jdi-stats]
 9: 0xf35fae v8::internal::Heap::AllocateRawWithRetryOrFailSlowPath(int, v8::internal::AllocationType, v8::internal::AllocationOrigin, v8::internal::AllocationAlignment) [mongosh mongodb+srv://<credentials>@HOST/jdi-stats]
10: 0xef777a v8::internal::Factory::NewFillerObject(int, bool, v8::internal::AllocationType, v8::internal::AllocationOrigin) [mongosh mongodb+srv://<credentials>@HOST/jdi-stats]
11: 0x1273006 v8::internal::Runtime_AllocateInYoungGeneration(int, unsigned long*, v8::internal::Isolate*) [mongosh mongodb+srv://<credentials>@HOST/jdi-stats]
12: 0x165b5d9  [mongosh mongodb+srv://<credentials>@HOST/jdi-stats]
Aborted (core dumped)

real    1m46.195s
user    0m55.926s
sys 0m3.521s
1 Answers

I cannot tell why this is such a pain.

I can. You are using wrong tool for the job. There is an old saying that screws fixed with a hammer hold better than nails fixed with a screwdriver.

Try mongoimport It's written in Go to handle massive loads of data in json,and with less flexibility in csv/tsv formats.

Related