Reduced API Get Requests when Multi Threaded

Blesson Xavier 0
Viewed 43

I am trying to call an get method multiple times of an API via 2 methods for a set of valid URLS endpoints

Method 1: One simple call multiple times via a loop

Method 2: Multiple Threads to call the API multiple times in each thread

While Calling these 2 methods I am getting HTTP:200 Responses for as many times I am calling the function

But when checking the logs of the Service I can see lesser number of calls ie counting from the message displayed everytime the get function is called via Method 2 (Multi-Threading)

Calls made are significantly higher than calls recieved in the logs when using multi-threading in python , which makes no sense to me though as all responses I recieve correctly

The service I am Calling is a URL Shortner Service and I am calling a shortened URL in the endpoint

Method 1:

with open('url_big_list.csv') as file_obj:
    reader_obj = csv.reader(file_obj)
        for row in reader_obj:
            for i in range(50):
                url='https://so2.in/'+row[0]
                print(url)
                headers = {'Content-Type': 'application/json'}
                response = requests.get(url,verify=False)

Method 2

def multiThreadRun(no_of_calls_per_thread,url_list):
    for i in range(no_of_calls_per_thread):
        url=random.choice(url_list)
        print(url)
        headers = {'Content-Type': 'application/json'}
        response = requests.get(url,verify=False)

no_of_threads = 10
thread_counter = 0
no_of_api_call_per_thread=100
temp = ""
thread_list=[]
while(thread_counter < no_of_threads):
    print("thread no :", thread_counter)
    temp = "t"+str(thread_counter)
    temp = threading.Thread(target=multiThreadRun(no_of_api_call_per_thread,url_list))
    thread_list.append(temp)
    thread_counter = thread_counter+1

Thanks for your Help in Advance

multithreading api python-3.x python
1 Answers

You need something like this after your while loop in your second example to make sure all the threads have finished before exiting the program:

for thread in thread_list:
    thread.join()

this will allow you to wait till all of the threads have completed. It does not matter what order they complete. If one is are already done when the join is called it will immediately return, otherwise it will wait. once you have made it through the list you will know they have all completed.

LhasaDad0
Related