How do I maintain multiple sessions to the same host while web-scraping?

Viewed 592

I am having a problem with my web-scraping code. Here is the gist of it:

I have a pool of proxies, and I initialize a session object for each proxy. After that, I pass the session object to a thread and that thread then web-scrapes a site (in this case, globenewswire.com). My problem lies in web-scraping the same site with multiple sessions at once. When I have multiple session objects that run in a multi-threaded environment, I get errors. When I run my session objects one at a time, I am completely fine.

import time
import requests
import threading

lock = threading.Lock()
proxy_pool = []  # List of 11 proxies with format: "0.0.0.0:0000"
session_pool = []


def print_lock(stuff):
    with lock:
        print(stuff)


def initialize_session(proxy):
    global session_pool
    proxiez = {"http": "http://" + proxy, "https": "http://" + proxy}
    s = requests.Session()
    s.mount('https://globenewswire.com', HTTPAdapter(pool_connections=1, pool_maxsize=11))
    s.proxies = proxiez
    while True:
        try:
            r = s.get('http://globenewswire.com/Search', headers={'headers': 'here'})
            if r.status_code == 200:
                break
            else:
                print_lock('r.status_code =', str(r.status_code), 'for proxy:', proxy)
        except Exception as e:
            print_lock('Error for proxy:', proxy + '. Error:', str(e))
        time.sleep(1)
    with lock:
        session_pool.append(s)
        print('Created Session now in session_pool with proxy: ' + proxy + '.')


startup_thread_list = []
for proxie in proxy_pool:
    startup_thread = threading.Thread(target=initialize_session, args=(proxie,))
    startup_thread_list.append(startup_thread)
    startup_thread.daemon = True
    startup_thread_list.append(startup_thread)
    startup_thread.start()
for thread in startup_thread_list:
    thread.join()
print('Set up all sessions.')


def remove_session():
    global session_pool
    while True:
        with lock:
            if len(session_pool) == 0:
                session_waited = 'yes'
            else:
                session = session_pool[0]
                session_pool.remove(session)
                break
        time.sleep(0.1)
    return session

while True:
    sesh = remove_session()
    doStuffwithSessionthread = threading.Thread(target=doStuff, args=(sesh,))
    doStuffwithSessionthread.daemon = True
    doStuffwithSessionthread.start()
    # Submit the session to only one thread
    # the thread then returns that session to the session_pool as its final task.
    time.sleep(4)

Interestingly enough, I tried opening a separate Python Interpreter for every proxy I had in my proxy pool and only passed one proxy to the pool of each interpreter. I effectively limited the amount of session objects to one per interpreter. I was able to run 11 interpreters side by side without any issue. However, if I try to run 11 sessions within 1 interpreter, I get more errors that status code 200s. This leads me to believe I am having a threading issue with the requests module or my own code.

Requests recommends that you only run one session object per thread. I know for a fact that I am only running at most one thread for every session object that I have, but for some reason my https requests are coming back as errors more often than not.

This code also works perfectly if you only have one session object making a request at any one time in the interpreter, but spits errors out like no tomorrow as soon as multiple session objects make requests at the same time.

This has been bugging me for a while, does anyone know what could be going on? Please let me know if I need to clarify anything else.

0 Answers
Related