MongoDB bulk.execute() taking too much time to insert unordered

Viewed 2409

I'm trying to feed a mongoDB(version 3.2.4) with arcs from google ngrams but it's taking too much time even with a decent configuration (MBP Mid 2014, 2.2 GHz Intel Core i7, 16gb).

For each one of the 8,9million rows in the original file, I create a doc and bulk.insert(doc); it into an unordered bulk.

After inserting 500 of them, I do bulk.execute() and repeat this process until all of them are added to the DB, which never happens, since it takes more than a minute per thousand doc's to be inserted and I sure don't have 8,9k minutes to wait for it.

here goes the code:

MongoClient.connect(url, function (err, db) {
if (err) {
  console.log('Unable to connect to the mongoDB server. Error:', err);
} else {
    //connected
    console.log('Connection established to', url);

    var bulk = db.collection('bigrams').initializeUnorderedBulkOp();

    const rl = readline.createInterface({
     input: fs.createReadStream(path+filename)
    });

    rl.on('line', function (line) {
        var stringArray = line.split("\t");
      var firstPart = stringArray[0]+'\t'+stringArray[1]+'\t'+stringArray[2]+"\t";

      var head_token = stringArray[0];
      var syntatic_ngram = stringArray[1].split(" ");
      var total_count = stringArray[2];
      var counts_by_year = line.replace(firstPart,'').split("\t");

      var doc = {
        "head token" : head_token,
        "syntatic ngram" : syntatic_ngram,
        "total count" : total_count,
        "counts by year" : counts_by_year
      };
      count++;
      bulkCount++;
      if (bulkCount == bulkSize) {
        console.log("BulkSize reached. Executing...");
        bulk.execute();
        bulkCount = 0;
      }
      console.log("bulk inserted count:"+count);
      bulk.insert(doc);
    });

    rl.on('end', function(){
      bulk.execute();
        db.close();
    });

  }
});

P.S.:The first 10k insertions, i.e., first 20 bulk.execute() are processes really fast <1m. After that processing time goes increasing. ): Any sugestions? Thank you.

1 Answers
Related