Multi-Threading BS4 scraper does not speed up the process

user_14861772 0
Viewed 59

I am trying to multithread my scraper so it runs faster. Currently I commented pagination so it finished faster for time measurement but it runs the same as the simple scraper that I did not use concurrent.futures.ThreadPoolExecutor. Instead it after I try to quit the script from executing using Crtl+c it seems to quit one process, but immediately after quitting ing it seems to continue the same scraper, and I have to stop it from executing again, so something changes, but not the speed, nor the data.

This is my scraper:

from bs4 import BeautifulSoup
import requests
import concurrent.futures

NUM_THREADS = 30

BASEURL = 'https://www.motomoto.lt'
HEADERS = {'User-Agent': 'Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/89.0.4389.82 Safari/537.36'}

page = requests.get(BASEURL, headers=HEADERS)
soup = BeautifulSoup(page.content, 'html.parser')

item_list = []

def main():
    with concurrent.futures.ThreadPoolExecutor(max_workers=NUM_THREADS) as executor:
        executor.map(parse_category, soup)


def parse_category(soup):
    for a in soup.find_all('a', class_='subcategory-name', href=True):
        nexturl = BASEURL + a['href']
        parse_subcategory(nexturl)

def parse_subcategory(url):
    subcategoryPage = requests.get(url, headers=HEADERS)
    soup = BeautifulSoup(subcategoryPage.content, 'html.parser')
    for a in soup.find_all('a', class_='subcategory-image', href=True):
        nexturl= BASEURL + a['href']
        parse_products(nexturl)

def parse_products(url):
    productsPage = requests.get(url, headers=HEADERS)
    soup = BeautifulSoup(productsPage.content, 'html.parser')
    for a in soup.find_all('a', class_='thumbnail product-thumbnail', href=True):
        nexturl = a['href']
        parse_item(nexturl)

    # this = soup.find('a', attrs={'class':'next'}, href=True)
    # if this is not None:
    #     nextpage = BASEURL + this['href']
    #     print('-' * 70)
    #     parse_products(nextpage)
        
        
def parse_item(url):
    itemPage = requests.get(url, headers=HEADERS)
    soup = BeautifulSoup(itemPage.content, 'html.parser')
    title = get_title(soup)
    price = get_price(soup)
    category = get_category(soup)
    
    item = {
        'Title': title,
        'Price': price,
        'Category': category
    }

    item_list.append(item)
    print(item)
    
def get_title(soup):
    title = soup.find('h1', class_='h1')
    title_value = title.string
    title_string = title_value.strip()
    return title_string

def get_price(soup):
    price = soup.find('span', attrs={'itemprop':'price'}).string.strip()
    return price

def get_category(soup):
    category = soup.find_all("li", attrs={'itemprop':'itemListElement'})[1].find('span', attrs={'itemprop':'name'}).getText()
    return category

if __name__ == "__main__":
    main()

Currently I am multithreading the first function, that uses the BS4 soup to gather the category links. How may I fix it to make it faster, even though it's using multiple functions?

web-scraping multithreading python
1 Answers

The signature of ThreadPoolExecutor.map is

map(func, *iterables, timeout=None, chunksize=1)

The executor processes iterables concurrently. If you have supplied multiple soups like executor.map(parse_category, [soup1, soup2, ...]) they will be processed in parallel. But since you have supplied only one soup, you are "doing one thing concurrently", which means there is no concurrency.

As you are calling parse_category only once, it worse not adding concurrency to it. Instead, you can parallelize parse_subcategory and parse_products like this:

...

def main():
    executor = concurrent.futures.ThreadPoolExecutor(max_workers=NUM_THREADS)
    parse_category(soup, executor)

def parse_category(soup, executor):
    executor.map(
        lambda url: parse_subcategory(url, executor),
        [BASEURL + a['href'] for a in soup.find_all('a', class_='subcategory-name', href=True)])


def parse_subcategory(url, executor):
    subcategoryPage = requests.get(url, headers=HEADERS)
    soup = BeautifulSoup(subcategoryPage.content, 'html.parser')
    executor.map(
        lambda url: parse_products(url, executor),
        [BASEURL + a['href'] for a in soup.find_all('a', class_='subcategory-image', href=True)])
        

def parse_products(url, executor):
    productsPage = requests.get(url, headers=HEADERS)
    soup = BeautifulSoup(productsPage.content, 'html.parser')
    executor.map(
        parse_item,
        # here you missed the `BASEURL`, I kept it as-is
        [a['href'] for a in soup.find_all('a', class_='thumbnail product-thumbnail', href=True)])
...

The remainder of the script is unchanged.

I didn't test it as the website seems inaccessible from my location. Reply if there's any bug.

user_93996180
Related