Efficient way to iterate over cursor object with Pymongo / MongoDB

user_12181414 0
Viewed 761

I prefilter 3 collections and build a single new collection from it. To do this i iterate over the cursor objects like this (monate, wochen, tage are justs lists containing relevant datetime objects:

monate_final = collection1.find({"NewDate": {"$in": list(monate)}})
wochen_final = collection2.find({"NewDate": {"$in": list(wochen)}})
tage_final = collection3.find({"NewDate": {"$in": list(tage)}})

master_list = [monate_final, wochen_final, tage_final]

for collection in master_list:
    for document in collection:
        self.target.insert_one(document)

The code works, but since the the final collections contains > 100 millions records, the process just takes forever. I did not find a more efficient way to do this. Building a pandas DataFrame and then use insert_many() does not work due to memory limitation. Can anyone help me?

pymongo python
2 Answers

Use bulk operations and break the bulk writes into chunks of, say, 50,000 as per this example. You can either use a counter (probably quicker) or just check len(updates):

from pymongo import InsertOne
updates = []
counter = 0

for collection in master_list:
    for document in collection:
        updates.append(InsertOne(document))
        counter += 1

        if counter > 50000:
            self.target.bulk_write(updates)
            counter = 0
            updates = []

# Update the final items after the cursor has exhausted
if len(updates) != 0:
    self.target.bulk_write(updates)
user_104906830

I don't have enough reputation to make this as a comment.

I have not validated this. But you could make an aggregation pipeline which filters your collections. And the final step in the pipeline is the $out operator to save all documents in the pipeline into the new collection.

https://docs.mongodb.com/manual/reference/operator/aggregation/out/

I'm not deep into the inner workings from MongoDB. But I would expect the pipeline to be run completely DB side which would greatly improve the performance of the operation.

Wish you luck solving this! (Don't forget to add your solution for other people :D )

user_110438490
Related