Is python selenium browser.quit() method processes safe?

Question

I am running multiple browsers using multiprocessing Pool. Each process will be opening and closing browsers over and over as well. The reason for each process to close the browser and open a new one is that the sites I am visiting will block with captcha on the second visit if not. If each process calls browser.quit() will it affect chrome instances running in other processes? I am having trouble with some sites failing when I know they are good URLs.

EDIT: Let me explain further. Selenium visits the site and I am returning the HTML for scraping. The error I receive in the log file:

Failed While scraping 
object of type 'NoneType' has no len()

example selenium code:


def get_page(url):
    browser = webdriver.Chrome(executable_path=r'chromedriver.exe')
    time.sleep(2)
    browser.get(url)
    
    # verify page
    if browser.current_url[-19:] == 'noResultsFound=true' or browser.current_url[-13:] == 'error404.html':
        browser.quit()
        return None
    else:
        html = browser.page_source
        browser.quit()
        return html

example scraping code:

def scrape(html):
    
    soup = BeautifulSoup(html, 'html.parser')
    search_items = soup.find('div', {'class': 'row product-grid results'})
    if search_items is not None:
        search_items = search_items.find_all('div', {'class': 'col-xs-12 col-xs-6 col-sm-4 col-md-3 text-center'})

    for i in range(len(search_items)):
        # scrape each search result

I can visit the URL and verify that the

s do exists but I am failing on the for loop by taking the length. My thought was either JavaScript is not fully loading on page before returning the page_source or another processes calling browser.quit() affects the other processes.

Is python selenium browser.quit() method processes safe?

0 Answers