I want to get 1 million records from a single REST API URL with pagination. Each page can get up to 100k records in approx 1 min. I want to make 10 post requests concurrently so that I can get all 1 million records in 1 min. I used requests library, ThreadPoolExecutor to make concurrent connections and get the data but it's taking a very long time to get the data.
My understanding of aiohttp or grequests is that they call the REST API asynchronously from a single thread and get the data while waiting for others to make network connections.
Sample code
def post_request(page_number):
response = requests.post(url, data = data, fetchPageNumber:page_number)
# logic to convert response.text pandas df and write to RDBMS
with concurrent.futures.ThreadPoolExecutor(10) as executor:
parallel_response = executor.map(post_requst, list(1,11)
Please let me know what is the best way to get the paginated data in parallel.