Use multiprocess with API requests and multiple for loops on Python

user_15673147 0
Viewed 397

I'm accessing an API to get specific public budget from Brazil. It needs to define year, month and page. I was successful to use for loops to get the info I want for the year of 2020, looping through months {j} and pages (str +1).

How can I parallelize the following (even better if I can turn it to a def function and using map)?

list1 = []

for i in tqdm(range(x)):
        for j in tqdm(range(1,13)):
            url = f'https://gatewayapi.prodam.sp.gov.br:443/financas/orcamento/sof/v3.0.1/empenhos?anoEmpenho=2020&mesEmpenho={j}&codOrgao=84&numPagina=' + str(i+1)
            headers = {"Accept": "application/json", "Authorization": "Bearer xxxxxxxxxxxxxx"}
            response = requests.get(url, headers = headers)
            list1.append(response.json())

df_final = pd.DataFrame()
for i in range(len(list1)):
    df_temp = pd.DataFrame(list1[i]['lstEmpenhos'])
    df_final = df_final.append(df_temp)

df_final
multiprocess user-defined-functions for-loop api python
1 Answers

One thought might be to take the code in your nested for loop and break it out into a function:

def get_data(pair):
    i, j = pair
    url = f'https://gatewayapi.prodam.sp.gov.br:443/financas/orcamento/sof/v3.0.1/empenhos?anoEmpenho=2020&mesEmpenho={j}&codOrgao=84&numPagina=' + str(i+1)
    headers = {"Accept": "application/json", "Authorization": "Bearer xxxxxxxxxxxxxx"}
    response = requests.get(url, headers = headers)
    return response.json()

Then you could use something like the ThreadPoolExecutor and map that against your values. You could make this a lot better, but extremely naïvely:

list1 = []
parameters = []

pool = ThreadPoolExecutor(workers=6)

for i in tqdm(range(x)):
    for j in tqdm(range(1,12)):
        parameters.append((i, j))

list1 = list(pool.map(get_data, parameters[0:x]))
user_51748250
Related