How to use apply_parallel on db calls

Viewed 14

I was using apply_parallel function from pandarallel library, the below snippet(Function call) iterates over rows and fetches data from mongo db. While executing the same throws me EOFError and a mongo client warning as given below

Mongo function:

def fetch_final_price(model_name, time, col_name):
    collection = database['col_name']
    price = collection.find({"$and":[{"Model":model_name},{'time':time}]})
    price = price[0]['price']
    return price

Function call:

final_df['Price'] = df1.parallel_apply(lambda x :fetch_final_price(x['model_name'],x['purchase_date'],collection_name), axis=1)

MongoClient config:

client = pymongo.MongoClient(host=host,username=username,port=port,password=password,tlsCAFile=sslCAFile,retryWrites=False)

Error:

EOFError: Ran out of input

Mongo client warning:

"MongoClient opened before fork. Create MongoClient only "

How to make db calls in parallel_apply??

0 Answers
Related