Python: Splitting a "for obj in reponse.json() loop" to request from the xth object forth

Viewed 35

Here is a bit of context, I have a program to gets data from an API. It does this in two requests one for the total amount of points and the second a request for each point in the data. These get appended into an array.

def fetch_details(url: str):
    response = requests.get(url)
    # Makes request call to get the data of detail
    # save_file(folder_path,GipodId,text2)
    # any other processe
    return response.json()


def fetch_data_points(url: str):
    limit_request = 1000
    # Placeholder for limit: please do not remove = 1000000000 -JJ
    folder_path_reset("api_request_jsons","csv","Geographic_information")
    total_start_time = start_time_measure()
    start_time = start_time_measure(
        'Starting Phase 1: First request from API: Data Points')
    response = requests.get(url,params={"limit": limit_request})
    end_time = end_time_measure(start_time, "Request completed: ")
    print(len(response.json()))
    time_estimated = end_time/len(response.json())
    print(time_estimated)
    end_time_measure(total_start_time, "End of Phase 1, completed in: ")
    return response.json()

def fetch_details_of_data_points(url: str):
    input_json = fetch_data_points(url)
    fetch_points_save(input_json)
    all_location_points_details = []
    amount_of_objects = len(input_json)
    total_start_time = start_time_measure()
    start_time = start_time_measure(f'Starting Phase 2: Second request from API: {str(amount_of_objects)} requested')
    #for i in tqdm(range(amount_of_objects),miniters=0.000000001):
    #    for obj in input_json: 
    #        all_location_points_details.append(fetch_details(obj.get("detail")))
    with tqdm(total=amount_of_objects) as pbar:
        for obj in input_json: 
            all_location_points_details.append(fetch_details(obj.get("detail")))
            pbar.update(1)

However I have noticed a certain flaw in my program I may have a solution for but I do not know how to implement. You see when the amount of data requested is massive (over more than 10.000 points) there can always happen a disconnect causing my program to fail. So as a solution I have would like this loop:

with tqdm(total=amount_of_objects) as pbar:
    for obj in input_json: 
        all_location_points_details.append(fetch_details(obj.get("detail")))
        pbar.update(1)

To be split a factor of a value i (or x) that is calculated by the following:

value y = 1000
value x = round(Amount of objects/y) --> Round because this needs to be rounded up no matter.

So lets say I have 145862 objects to request details from by my formula that is suppose to be 14.5 rounded up 15 sessions.

So 1 session request the first 1000 obj, starting from obj 1 and ending at 1000. The next session starts requesting from obj 2001. Next sessions starts from obj

So this is technically this:

i = 0
while i < x
        for obj (starting from i + 1 object ending at 1*y ) in input_json: 
            all_location_points_details.append(fetch_details(obj.get("detail")))
            i += 1

Thing is the part of I do not know how to program this. Can anyone help me with this?

0 Answers
Related