BeautifulSoup, Selenium and Python, parsing by a tag

user_7290496 0
Viewed 125

I'm trying to parse data from this website

https://findrulesoforigin.org/home/compare?reporter=392&partner=036&product=020130010

In particular, I am trying to get the data under Criterion(ITC). The text I want says CC+ECT

The information I want in html appears to be 

<a class= js-glossary data-leg= "CC+ECT">

I'm new to web scraping and I tried the techniques taught in the tutorial but they didn't work. I heard about Selenium and tried this out too. However, this code didn't work either.

from selenium import webdriver
from bs4 import BeautifulSoup
import requests

driver = webdriver.Firefox(executable_path = r"D:\Python work\driver\geckodriver.exe")
driver.get(r"https://findrulesoforigin.org/home/compare?reporter=392&partner=036&product=020130010")
html = driver.page_source
soup = BeautifulSoup(html, 'lxml')
data = soup.find_all("a", attrs= {"class":"js-glossary"})

The code results in an empty list. I also read that I can pull out the data by treating the soup tag like a dictionary. in this case

data["data-leg"]

Am I on the right track or am I way off?

webdriverwait selenium-webdriver selenium beautifulsoup python
2 Answers

The text you're trying to get generated dynamically by JavaScript. To get it you need to wait for its appearance:

from selenium import webdriver
from selenium.webdriver.support.ui import WebDriverWait

driver = webdriver.Firefox(executable_path = r"D:\Python work\driver\geckodriver.exe")
driver.get(r"https://findrulesoforigin.org/home/compare?reporter=392&partner=036&product=020130010")
text = WebDriverWait(driver, 5).until(lambda driver: driver.find_element_by_xpath('//div[.="criterion(itc)"]/following-sibling::div').text)
print(text)
#  'CC + ECT'
user_45495540

Seems you were pretty close. You may not even require Beautiful Soup if you are using Selenium. Using Selenium you need to induce WebDriverwait for the desired element to be visible and you can use the following solution:

  • Code Block:

    from selenium import webdriver
from selenium.webdriver.common.by import By
from selenium.webdriver.support.ui import WebDriverWait
from selenium.webdriver.support import expected_conditions as EC

driver = webdriver.Firefox(executable_path = r'C:\Utility\BrowserDrivers\geckodriver.exe')
driver.get(r"https://findrulesoforigin.org/home/compare?reporter=392&partner=036&product=020130010")
print(WebDriverWait(driver, 20).until(EC.visibility_of_element_located((By.XPATH, "//div[@class='lbl' and text()='criterion(itc)']//following::div[1]/a"))).get_attribute("innerHTML"))

  • Console Output:

                                    CC + ECT
user_74294470
Related