I am experimenting Python multiprocessing and I am surprised to see counter intuitive results while using locks.
My assumption was that while using shared memory, if we use a lock to ensure only one process can access the shared object at a time, all the other processes will wait for the locking process to release the lock before reading/writing on the shared object. So although this ensures data integrity and eliminates race conditions, this will also inevitably cause all but one process to wait on a locking process. As a result the overall time of execution will be longer than while not using locks. If no locks are used, none of the processes will wait on each other and so will finish faster but will also corrupt the data.
With this understanding in mind, I wrote a quick program to find out exactly how much slower is it to use a lock, but surprisingly it seems using a lock is actually significantly faster.
Can you explain why is it faster to use a lock? Is something wrong with my understanding of how locks work? Or is there something wrong with the way I’ve setup the experiment (code below).
If lock.acquire() and lock.release() are commented out, so NOT using lock, it takes 75.85 seconds,
If lock.acquire() and lock.release() are Uncommented, so using lock, it takes 57.17 seconds
I am using Python 3.7,
Running on Intel i7 8550U, 4 physical cores, (8 threads with hyperthreading)
import os
import multiprocessing as mp
from timeit import default_timer as timer
def func(n, val, lock, add):
for i in range(n):
lock.acquire() # uncomment to use lock
if add:
val.value += 1
else:
val.value -= 1
lock.release() # uncomment to use lock
if i % (n/10) == 0:
print(f"Process: {os.getpid()}, {(i / n * 100):.2f}%")
if __name__ == "__main__":
start = timer()
lock = mp.Lock()
n = 2000000
val = mp.Value('i', 100)
arr = mp.Array('i', 10)
p1 = mp.Process(target=func, args=(n, val, lock, True))
p2 = mp.Process(target=func, args=(n, val, lock, False))
p1.start()
p2.start()
p1.join()
p2.join()
elapsed = timer() - start
print("Time elapsed = {}".format(elapsed))
print(f"Val = {val.value}")