I want to process a list of 1 million documents in Python Concurrent Future. But I see it consumes lot of time than a normal synchronous approach. Below is the sample code snippet with detail.
Code to create a list of dictionary
data = []
for i in range(1, 1000000):
data.append({"_id":i, "name":"MaxPayne "+str(i)})
Helper function to print text
def printText(d):
return d
Synchronous way
for i in data:
x=printText(i["name"])
print(x)
Sychronous way takes 8 seconds to process entire list and print the data
Concurrent Future Processpool
with concurrent.futures.ProcessPoolExecutor() as executor:
results = [executor.submit(printText, d["name"]) for d in data]
for f in concurrent.futures.as_completed(results):
print(f.result())
Concurrent Future way takes 300 seconds to process entire list and print the data
May I know what is the issue. Can any one help me fixing this bug. Thanks in Advance