Python: How to parallel get requests to minimize timing?

user_17680006 0
Viewed 83

I'm very new to programming and to stackoverflow rules in particular.

Please let me know if I missed any important information to indicate in my question.

I'm looking for a simple solution to parallel my get requests using selenium module on Python.

I have a list of 10,000 web links of restaurants on tripadvisor.com. I want my script to get html data from each link and find a text under a certain html class name. In the end my script should take each text grabbed and put it in a dictionary next to the link from the list.

Sending get requests sequentially link by link will take crazy amount of time.

How can I parallel these requests to minimize timing?

Now my code looks like this:

from selenium import webdriver

# PATH = 'path to driver'

driver = webdriver.Chrome(executable_path=PATH)

nans_dict = {}

for i in nans_list:
    a = []
    driver.get('https://www.tripadvisor.com' + i)
    for element in driver.find_elements_by_class_name('drUyy'):
        a.append(element.text + ' ')
        nans_dict[i] = a

Thank you for the comprehensive answers in advance.

get-request selenium parallel-processing python
1 Answers

Here's one way of doing this.

Create a dictionary whose keys are the significant parts of the Tripadvisor URL and whose values are a list. The text associated with the 'drUyy' class is appended to the relevant value. Thus, when complete, the dictionary has all the relevant text associated with the page where it was found.

from selenium import webdriver
from selenium.webdriver.common.by import By
from threading import Thread

restaurants = {'Restaurant_Review-g186338-d783897-Reviews-Indian_Room-London_England.html': [],
'Restaurant_Review-g186338-d10460592-Reviews-Bonoo_Indian_Tapas-London_England.html': [],
'Restaurant_Review-g186338-d13544747-Reviews-Amrutha_Lounge-London_England.html': []}

def process(r):
    options = webdriver.ChromeOptions()
    options.add_argument('--headless')

    with webdriver.Chrome(options=options) as driver:
        driver.get(f'https://www.tripadvisor.co.uk/{r}')
        for a in driver.find_elements(By.CLASS_NAME, 'drUyy'):
            restaurants[r].append(a.text)

def main():
    thread_list = []
    for r in restaurants:
        t = Thread(target=process, args=(r,))
        t.start()
        thread_list.append(t)
    for t in thread_list:
        t.join()
    print(restaurants)

if __name__ == '__main__':
    main()
user_175803810
Related