How to pull hourly data from a URL in Python?

user_16565901 0
Viewed 42

I'm querying data in JSON format, but I want it to continue pulling every time updated data is added. I also insert this data to the Postgresql database. But since data is added to the url hourly, instead of waiting for the database to fill up, I want it to quickly pull the last 2-3 months' data at the first time with a loop and then continue to shoot every hour. How can I do that?

parameters = {
  "StationId": "377e1216-bcc7-42c0-aad8-4d5b3d602b78",
  "StartDate": "12.01.2022%2000:00:00",
  "EndDate": "12.01.2022%2001:00:00 }
payload = {}
headers= {}
url="https://api.ibb.gov.tr/havakalitesi/OpenDataPortalHandler/GetAQIByStationId"

req = requests.get(url, params = parameters)
if req.status_code == 200:
    decodeUrl = unquote(istek.url)
    response = requests.get(decodeUrl,headers=headers, data = payload)
    result = json.loads(response.text)
    print("Success")
else:
    print("Wrong.")
postgresql python json
1 Answers

You could start by making the parameters variable, so you can use them in every call. Something like this:

parameters = {
  "StationId": "377e1216-bcc7-42c0-aad8-4d5b3d602b78",
  "StartDate": start_date,
  "EndDate": end_date }

Then split the functionality to trigger the calls into 2. 1 "script" (for lack of better terminology) is triggered once to populate the database. Get the current date and subtract 3 months, then put that into the parameters.

Your second script would do the same but instead take the previous hour and input that into the parameters. Then (depending on your OS) run a cron-job or scheduled task where you fire off the second script.

user_74363210
Related