Dynamically store the API data

user_11603094 0
Viewed 199

I have to extract data from API's and return the report with the required info. For eg:

request_data = {'url1 : https://abcd.com','url2 : https://dfgh.com','url3 : https://hjkl.com',column : (name,Ecode,salary,status)}

x1 =  '{ "name":"John", "age":30, "city":"New York"}'

x2 = '{ "name":"John", "age":30, "Ecode":"4567"}'

x3 = '{ "name":"John", "salary":30k, "Status":"Yes"}'

here request_data is the get request that I receive through a messaging queue and x1,x2,x3 are the values which we get after running the get request from python. Once we get the data I convert it to pandas dataframe and use pandas dataframe merge to create a new dataframe.

But the number of url's wont be 3 in the get request. It would be 2,5,1, etc. So we cannot explicitly name the variables as x1,x2,x3 as the variables to store data.

I want to make code more dynamic so that any number of url request we get from message queue, we need to get data from those url and create a new dataframe with the required columns.

Can someone please help me here or point to some reference link.

pandas api python json
1 Answers

You could start by creating an empty dataframe called cumulative_df in which you could collect the responses of the different API calls.

Then, loop through the list of URLs; within the loop, for each URL:

  1. Make the API call (e.g., by using the requests library).
  2. Convert the JSON response to a dataframe (e.g., by using pandas.json_normalize)
  3. Add the dataframe created by json_normalize to cumulative_df (e.g., by using pandas.concat)

This could be done for any number of URLs in your list.

Edit: if the dataframes cannot be concatenated/appended, you could use a counter to name the variable and assign the dataframe to it inside of the loop. At first, 'x'+str(i) will result in x1. Then increment the counter by 1, and the next instance of the loop would create x2, and so on.

user_20545300
Related