Pandas DataFrame Rest Call on each row

user_2771169 0
Viewed 817

My Pandas data frame has one million rows. I have to call a rest API call inside a function on each row and need to capture the response. Each rest call taking 2 sec of time on an average. But the following cases which I tried are very slow

Case 1: apply

def predict(x):
     res = request("XYZ")
     return res.json()

df['response_value'] = df.apply(lambda x:predict(x['request_filed']),axis=1)

Case 2: Vectorisation instead of apply

def predict(x):
     l = []
     for each in x
         l.append(request("XYZ"))
     return l

df['response_value'] = predict(df['request_filed']

Case 3: Paralleled apply

def predict(x):
     res = request("XYZ")
     return res.json()

df['response_value'] = df.parallel_apply(lambda x:predict(x['request_filed']),axis=1)

Is their any better way to speed up the process?

pandas multithreading python
1 Answers

It can't be faster than the cumulative time taken for each API call. So my suggestion is to first do those:

responses = [request(x).json() for x in df['request_filed'].values]

Then add them to your df:

df['response_value'] = responses

Which will be very fast. That seems like the fastest you will get.

user_62207590
Related