Can't find data while scraping site using BeautifulSoup or Selenium

user_14266974 0
Viewed 83

I am trying to scrape a site for link to the newest factsheet. I've tried using Selenium and BeautifulSoup, however each time I am unable to find the link using the tools. For instance when checking the output using Soup I get nothing from the part. Any suggestions?

Link to site scraped site

Using selenium:

#BIOG
from selenium.webdriver.common.by import By
from selenium import webdriver
from selenium.webdriver.chrome.options import Options
options = Options()
options.headless = True

driver = webdriver.Chrome(options=options)
driver.get('https://www.biotechgt.com/performance/monthly-factsheets')
html = driver.page_source

driver.find_elements(By.XPATH, '/html/body/div/main/section/div/div/div/div/div[2]/div/div[1]/div[2]/div/table/tbody[1]/tr[2]/td/a')
web-scraping selenium beautifulsoup
2 Answers

You have page source

html = driver.page_source

but you are not using it in soup at all.

so change that :

soup = BeautifulSoup(driver.page_source, "lxml")

As far as Selenium is concerned :

You can use below css selector :

a[href^='https://www.biotechgt.com/download']

in code

ele = driver.find_element(By.CSS_SELECTOR, "a[href^='https://www.biotechgt.com/download']")

then you can do

ele.click() or any other stuff with web element.

Update 1:

driver.maximize_window()
driver.implicitly_wait(30)
driver.get("https://www.biotechgt.com/performance/monthly-factsheets")
wait = WebDriverWait(driver, 10)
wait.until(EC.element_to_be_clickable((By.XPATH, "//a[text()=' Allow all cookies ']"))).click()
driver.execute_script("var scrollingElement = (document.scrollingElement || document.body);scrollingElement.scrollTop = scrollingElement.scrollHeight;")
wait.until(EC.element_to_be_clickable((By.XPATH, "//button[text()='Accept']"))).click()
ActionChains(driver).move_to_element(wait.until(EC.visibility_of_element_located((By.CSS_SELECTOR, "a[href^='https://www.biotechgt.com/download']")))).perform()
for link in driver.find_elements_by_css_selector("a[href^='https://www.biotechgt.com/download']"):
    print(link.get_attribute('href'))
user_94440730

To get all download links from the page, you can use next example:

import requests
from bs4 import BeautifulSoup

url = "https://www.biotechgt.com/performance/monthly-factsheets"

soup = BeautifulSoup(
    requests.get(url, cookies={"dp-disclaimer": "APPROVED"}).content,
    "html.parser",
)

for a in soup.select("a.gtm-downloads:has(.btn-download)"):
    print(a["href"])

Prints:

https://www.biotechgt.com/download_file/force/191/209
https://www.biotechgt.com/download_file/force/187/209
https://www.biotechgt.com/download_file/force/185/209
https://www.biotechgt.com/download_file/force/184/209

...
user_100359850
Related