Import more than 1 json file using mongoimport

user_850234 0
Viewed 35570

I am new to mongodb and want to know about importing a json file from one server to another. I tried the following command mongoimport -d test -c bik check.json and it works fine for me. Now i want to know when there are multiple json files how do i import all of them at a single go. I could not find any related document where it is written this is not possible. Please help me is this possible and how

mongoimport mongodb json
13 Answers

Linux:

> cat one.json two.json > three.json

> mongoimport --db foo --collection baz --file three.json"

Or, all files in the folder :

> cat *.json > big.json

> mongoimport --db foo --collection baz --file "big.json"
user_62531000

One line solution:

for /F %i in ('dir /b c:\files\*.json') do mongoimport.exe /d db /c files /file c:\file\%i

user_8543010

Not sure whether it's a new feature, but mongoimport now can actually read from stdin. So what one can do to import multiple JSON files is as simple as

cat *.json | mongoimport --uri "mongdb://user:password@host/db?option=value" --collection example

I'm using mongodb-tools v4.2.0 btw.

UPDATE

mongodbimport can potentially consume a high amount of memory which may cause the program to be kill by system OOM. My machine's got 32GB RAM and this happened consistently when I tried to import ~10GB of data which is stored in RAM disk.

To divide a relatively large job into batches:

#!/usr/bin/env bash

declare -a json_files=()
for f in *.json; do
    json_files+="$f"
    if [[ "${#json_files[@]}" -ge 1000 ]]; then
        cat "${json_files[@]}" | mongoimport --uri="mongodb://user:pass@host/db" --collection=examples -j8 #--mode=upsert --upsertFields=id1
        json_files=()
    fi
done
user_24872270

I used the solutions here to add a shell function to my bash profile for doing this quickly.

My example depends on the mongo export outputting each collection as a file with the collection name and .metadata.json extension.

function mimport() { for filename in *; do collection="${filename%.metadata.json}"; mongoimport --db $1 --collection $collection --file $filename; done }

Use in the path of the export files, passing the DB name to the command...

mimport my_db

Will load all collections into the DB at localhost.

user_3934760

Python:

from pathlib import Path
import subprocess

jsons_folder = "./out/"
mongodb_host = "172.22.0.3"
mongodb_port = "27017"
mongodb_user = "root"
mongodb_password = "1234"

for f in Path(jsons_folder).glob("*.json"):
    cmd = [
        "mongoimport", "-h", mongodb_host, "-p", mongodb_port, "--authenticationDatabase", "admin",
        "-u", mongodb_user, "-p", mongodb_password,
        "--db", "callscoring", "--collection", "scoring_result_entry", str(f.absolute())
    ]
    subprocess.run(cmd)
user_13249630
Related