pymongo - Why is Multiprocessing not faster

Question

I want to load 100.000 documents from a MongoDB into RAM. (1 document ~ 1,6kb) This takes about 2500ms with find().

The database and the Python script both have enough RAM and use as much as they need.

When I use multiprocessing it takes just as long as without. Strange is that the single access becomes shorter when I have less workers, why is that?

Code:

from multiprocessing import *
import time
import datetime
from pymongo import MongoClient


def worker(num):
    start = time.monotonic_ns()
    client = MongoClient(maxPoolSize=10000)
    db = client.Test_DB
    col = db.TEST_100000

    value_list = list(col.find().limit(100))
    
    end = time.monotonic_ns()
    print("Worker:" + str(num) + " End Timestamp: ", datetime.datetime.now() , " Time needed: ", (end-start)/1000000)



if __name__ == '__main__':  
    print("Start Timestamp: ", datetime.datetime.now())
    start = time.monotonic_ns()

    nums = [0,1,2,3,4,5,6,7,8,9]

    pool = Pool(processes=10)
    pool.map(worker, nums)
    pool.close()
    pool.join()


    end = time.monotonic_ns()
    print("End Timestamp: ", datetime.datetime.now(), " Time needed: ", (end-start)/1000000)

Output with 10 Workers:

Worker:1 End Timestamp:  2022-09-12 11:21:35.167935  Time needed:  1344.0
Worker:0 End Timestamp:  2022-09-12 11:21:35.236934  Time needed:  1406.0
Worker:3 End Timestamp:  2022-09-12 11:21:35.303513  Time needed:  1469.0
Worker:5 End Timestamp:  2022-09-12 11:21:35.340393  Time needed:  1500.0
Worker:2 End Timestamp:  2022-09-12 11:21:35.346391  Time needed:  1500.0
Worker:6 End Timestamp:  2022-09-12 11:21:35.359274  Time needed:  1500.0
Worker:7 End Timestamp:  2022-09-12 11:21:35.366889  Time needed:  1516.0
Worker:4 End Timestamp:  2022-09-12 11:21:35.368338  Time needed:  1531.0
Worker:8 End Timestamp:  2022-09-12 11:21:35.375989  Time needed:  1516.0
Worker:9 End Timestamp:  2022-09-12 11:21:35.378788  Time needed:  1516.0
End Timestamp:  2022-09-12 11:21:35.513204  Time needed:  2125.0

Output with 2 Workers:

Worker:0 End Timestamp:  2022-09-12 11:22:04.293868  Time needed:  328.0
Worker:2 End Timestamp:  2022-09-12 11:22:04.326558  Time needed:  359.0
Worker:1 End Timestamp:  2022-09-12 11:22:04.659217  Time needed:  375.0
Worker:3 End Timestamp:  2022-09-12 11:22:04.699550  Time needed:  359.0
Worker:4 End Timestamp:  2022-09-12 11:22:05.049932  Time needed:  391.0
Worker:6 End Timestamp:  2022-09-12 11:22:05.088083  Time needed:  375.0
Worker:5 End Timestamp:  2022-09-12 11:22:05.397071  Time needed:  343.0
Worker:7 End Timestamp:  2022-09-12 11:22:05.450128  Time needed:  359.0
Worker:8 End Timestamp:  2022-09-12 11:22:05.703320  Time needed:  297.0
Worker:9 End Timestamp:  2022-09-12 11:22:06.004075  Time needed:  297.0
End Timestamp:  2022-09-12 11:22:06.233088  Time needed:  2500.0

Update

In fact, multiprocessing is faster.

If I put the script in a Docker container, just like my DB, multiprocessing is faster, just like the normal query. 100k ~ 200ms. 10k ~ 20ms.

Running the same python script on my PC I get the bad results from above.

This is weird, does anyone have an explanation for this? Does a docker container get RAM allocated and therefore needs less swapping?

Rachid K. · Answer

If the single access decreases with less workers, this is likely due to some sort of mutual exclusion lock. The database that you are accessing may have a mutex which allows one access at a time and not multiple concurrent ones. Hence, the more you have workers trying to access it, the more contention on the lock you get. In other words, the access to the database may be serialized.

Is there a way to allow multiple readers at a time that would avoid this sort of mutual exclusion?

pymongo - Why is Multiprocessing not faster

Update

1 Answers