Insert 271 million records to MongoDB

Viewed 1209

I have 271 million records, line by line in a text file that I need to add to MongoDB, and I'm using Python and Pymongo in order to do this.

I first split the single file containing the 271 million records into multiple files containing each 1 million lines, and have written the current code to add it to the database:

import os
import threading
from pymongo import MongoClient


class UserDb:
    def __init__(self):
        self.client = MongoClient('localhost', 27017)
        self.data = self.client.large_data.data


threadlist = []

def start_add(listname):
    db = UserDb()
    with open(listname, "r") as r:
        for line in r:
            if line is None:
                return
            a = dict()
            a['no'] = line.strip()
            db.data.insert_one(a)
    print(listname, "Done!")


for x in os.listdir(os.getcwd()):
    if x.startswith('li'):
        t = threading.Thread(target=start_add, args=(x,))
        t.start()
        threadlist.append(t)

print("All threads started!")


for thread in threadlist:
    thread.join()

This starts as many threads as there are files, and adds every line to the db as it goes through it. The bad thing is that after 3 hours it had only added 8.630.623.

What can I do to add the records faster?

One row of data is just 8 digits: (e.g. 12345678)

1 Answers

Rather than doing multiple insert_ones, take a look a the bulk write operators.

Play around a little to see what works best; I've found batches of 10,000 work for me, but this is hardware dependant.

Related