I have a very large data set (71M rows - ~7GB csv file) that I have loaded into a data-frame in pandas.
I need to send each row in the data frame in a post request call to an API, and then store the response into another data-frame which is exported later and analyzed
Currently my code looks like this
##read data from csv
for row in data.itertuples(index=True, name='Pandas'):
##There is an if-else ladder to create a header depending on the type of values in a row
##Code to create a json payload
r = requests.post(url, data=json.dumps(payload), headers=headers)
t = json.loads(r.text)
## A try and except block to add the data sent via header and payload + the response from API call back into a new data-frame. (exception is in case there is no response from the API)
##write the data back to csv
The API can handle a QPS (queries per sec) of about 50,000 plus, however this method of executing just hits it with roughly 11 QPS. In my previous test, I split a smaller set of data (about 7M rows) 4 times,and ran 4 different Jupyter notebooks with the same code , so effectively reaching a QPS of about 44-50 and ran the code for 24 hours.
Note: I don't wish to hit it with that much QPS as it is a production API , I've been given an allowance of about 10k QPS
Since I now need to run it on a much larger set, is there a way this can be done in Python? Is splitting the data into 4 chunks and running them together equivalent to processing the code in parallel ?
Maybe I am going about this all wrong and there is some other way ? - my experience with python is mainly for analytics and data science work (numpy, pandas etc) - so this was the only approach that came to my mind.
I'm using a system with Intel xenon e5-2690 v2 processor (20 cores) and 128 GB RAM, so I think it ought to be capable to handle this since in my previous execution, it hardly broke a sweat in terms of resource utilization.
Any help to point me in the right direction would be appreciated.
EDIT: All the suggestions pointed me to aiohttp, however because I was running out of time, and I had made headway with multiprocessing's pool I went ahead with that. Added a few extra lines of code
if __name__=='__main__':
##read data
data_split = np.array_split(data,20)
p = Pool(20)
p.map(apicall, data_split)
The 'apicall' function is essentially the same as above bit of code (the for loop section)
It runs fine for about a dataset of 10,000. However if I scale it up to 100,000 or more, I get an error and I get stuck again
OSError: [WinError 10048] Only one usage of each socket address (protocol/network address/port) is normally permitted
Edit2: Understood why I was getting the above error. Looks like async is the only way to go.