scrape element not visible in page source

user_863713 0
Viewed 520

I'm trying to scrape a website (https://harleytherapy.com/therapists?page=1) that looks like it's been generated by Javascript and the element I'm trying to scrape (the lu with id="downshift-7-menu") doesn't appear on the "Page source" but only after I click on "Inspect element".

I tried to find a solution here and so far this the code I was able to come up with (a combination of Selenium + Beautiful soup)

import requests
from bs4 import BeautifulSoup
from selenium import webdriver
from selenium.webdriver.common.desired_capabilities import DesiredCapabilities
import time

url = "https://harleytherapy.com/therapists?page=1"

options = webdriver.ChromeOptions()
options.add_argument('headless')
capa = DesiredCapabilities.CHROME
capa["pageLoadStrategy"] = "none"
driver = webdriver.Chrome(chrome_options=options, desired_capabilities=capa)
driver.set_window_size(1440,900)
driver.get(url)
time.sleep(15)

plain_text = driver.page_source
soup = BeautifulSoup(plain_text, 'html')
therapist_menu_id = "downshift-7-menu"
print(soup.find(id=therapist_menu_id))

I thought that allowing Selenium to wait for 15 seconds would make sure that all elements are loaded but I still can't find any element with id downshift-7-menu in the soup. Do you guys know what's wrong with my code?

web-scraping selenium beautifulsoup python
1 Answers

The element with ID downshift-7-menu is loaded only after opening the THERAPIST dropdown menu, you can do it by scrolling it into view to load it and then clicking on it. You should also consider replacing sleep with explicit wait

from selenium.webdriver.common.by import By
from selenium.webdriver.support.ui import WebDriverWait
from selenium.webdriver.support import expected_conditions as EC


wait = WebDriverWait(driver, 15)

# scroll the dropdown into view to load it
side_menu = wait.until(EC.visibility_of_element_located((By.CLASS_NAME, 'inner-a377b5')))
last_height = driver.execute_script("return arguments[0].scrollHeight", side_menu)
while True:
    driver.execute_script("arguments[0].scrollTo(0, arguments[0].scrollHeight);", side_menu)
    new_height = driver.execute_script("return arguments[0].scrollHeight", side_menu)
    if new_height == last_height:
        break
    last_height = new_height

# open the menu
wait.until(EC.visibility_of_element_located((By.ID, 'downshift-7-input'))).click()

# wait for the option to load
therapist_menu_id = 'downshift-7-menu'
wait.until(EC.presence_of_element_located((By.ID, therapist_menu_id)))
print(soup.find(id=therapist_menu_id))
user_51680110
Related