Mongodb cursor iteration slow

user_4398653 0
Viewed 420

I am querying data from mongodb in chunks having 100M + records in total. And then want to process those records within a function.

My query

cursor = collection.find({'facial_task':False}).sort("_id", -1).skip(1000000).limit(10000)

When I iterate over the cursor, then it is taking too much time and gets stuck a lot even I limit the number of results up to 10. I even retrieved the results using batch size but when processing the loop over it; it takes too much time.

My loop is like this

for dd in cursor:
            ab = threading.Thread(target=insert_func, args=(dd,)) 
            ab.start()
            main_threads.append(ab)
            if len(main_threads) >= 5000:
                print("****Joining Main Thread***")
                for ii in main_threads:
                    ii.join()
                main_threads = []
pymongo mongodb python
1 Answers

If the slowdown occurs from the very first iteration, then it must have to do with the cursor itself.

Mongodb cursor is not like a 'chached' list of the entire collection; it queries the db one chunk at a time as you iterate over it. You can try to cast that cursor into a list and see if the iteration shows a significant speed-up.

user_153377350
Related