how to loop in python API with restriction on LIMIT and can provide offset

user_17715895 0
Viewed 666

I am new to python API. I need help in calling API data from python where we have limit of 20000 rows per page. Yes off course we can apply offset. But I am looking for a loop to extract complete data. Below is the python script. Sry I am new and not having much experience in writing the script.

querystring1 = {"limit":"20000","month":"122021","offset":"20000","dateformat":"X"}    
querystring2 = {"limit":"20000","month":"122021","offset":"40000","dateformat":"X"}    
querystring3 = {"limit":"20000","month":"122021","offset":"60000","dateformat":"X"}    

response = requests.request("GET", BASE_URL, headers=headers, params=querystring)    
response1 = requests.request("GET", BASE_URL, headers=headers, params=querystring1)    
response2 = requests.request("GET", BASE_URL, headers=headers, params=querystring2)    
response3 = requests.request("GET", BASE_URL, headers=headers, params=querystring3)    

data = json.loads(response.text)    
data1 = json.loads(response1.text)    
data2 = json.loads(response2.text)    
data3 = json.loads(response3.text)    

db = pd.DataFrame(data)    
db1 = pd.DataFrame(data1)    
db2 = pd.DataFrame(data2)    
db3 = pd.DataFrame(data3)    

month = 'DEC2021_UAE'    

csvdata = db.to_csv(month+'_1.csv',index=True,encoding="utf-8")    
csvdata1 = db1.to_csv(month+'_2.csv',index=False,encoding="utf-8")    
csvdata2 = db2.to_csv(month+'_3.csv',index=False,encoding="utf-8")    
csvdata3 = db3.to_csv(month+'_4.csv',index=False,encoding="utf-8")```   

at the end I have 4 different csv files however I want to able to loop through the API and should extract the data in file rather then 4 files.

Any Help Please...
api python
1 Answers

Loop over your offset's, make an API call and store the data and then combine the DataFrames in one operation using pd.concat.

offsets = ["20000", "40000", "60000"]
api_data = []

for offset in offsets:
    qs = {"limit":"20000","month":"122021","offset":offset,"dateformat":"X"}
    response = requests.request("GET", BASE_URL, headers=headers, params=qs)
    data = resp.json()
    api_data.append(data)

month = 'DEC2021_UAE'
csv_data = pd.concat((pd.DataFrame(data) for data in api_data), ignore_index=True)
csv_data.to_csv(month)
user_68178350
Related