find method not extracting already present div tag

user_8995741 0
Viewed 59

I am trying to scrape this webpage

I want to get the content from the following table:

enter image description here

The Statistics table content is present inside: div tag having "srl-tab srl-tab-handball-playerstats sr-widget sr-widget-level-0 sr-handball-playerstats sr-normal" as class-name

I am using the following code:

from selenium import webdriver
from bs4 import BeautifulSoup

link = "https://www.liquimoly-hbl.de/en/import/games/season-2020-2021/bundesliga/21--gameday--rhein-neckar-loewen---eulen-ludwigshafen/"

driver = webdriver.Chrome("path-to-my-chromedrivers")
driver.get(link)

driver.switch_to.frame("iframe-23400665")

page_source = driver.page_source

driver.close()

soup = BeautifulSoup(page_source, "html5lib")

a = soup.find("div", {"class": "srl-tabs-wrapper srl-flex-child"}).find("div", {"srl-tabs srl-flex"}).find("div", {'class': "srl-tabs-content-wrapper srl-flex-child"}).find("div", {"class": "srl-tabs-content"})

print(a.find("div", {"class": "srl-tab srl-tab-handball-playerstats sr-widget sr-widget-level-0 sr-handball-playerstats sr-normal"}))

The output of the print statement is giving me "None" value, since there is a div tag present in the website with the same class name I don't know why I am not getting the required output.

What is the thing that I have to change in my code? Or am I doing something wrong here?

Edit:

I think the only way to get the Statistics table value from the page-source is by first clicking on the Statistics button and then extracting the page-source. I do not know how to click on the Statistics button,

driver = webdriver.Chrome('path-to-my-chromedrivers')
driver.get(link)

driver.switch_to.frame("iframe-23400665")

page_source = driver.page_source

x = driver.find_elements_by_xpath("//div[@class='srl-tab']")
for i in x:
    if i.get_attribute("data-widget") == "handball.playerstats":
        print(i.get_attribute("class"))
        print(i.get_attribute("data-widget"))
        driver.execute_script("arguments[0].click();", i)
        print(i.get_attribute("class"))
        page_source_2 = driver.page_source
        break

driver.close()

Output:-

srl-tab
handball.playerstats
srl-tab selected

I am now able to click on the Statistics button (see class srl-tab changes to srl-tab selected)

But still no luck with getting the Player Total table.

selenium css html python
3 Answers

Go to page, click the popup, move to iframe, click on statistics and wait. Pass the source to Beautifulsoup and do what you want.

driver.get("https://www.liquimoly-hbl.de/en/import/games/season-2020-2021/bundesliga/21--gameday--rhein-neckar-loewen---eulen-ludwigshafen/")
driver.find_element(By.XPATH,"//button[.='ALLE AKZEPTIEREN']").click()
driver.switch_to.frame("iframe-23400665")
driver.find_element(By.XPATH,"//div[.='Statistics']").click()
time.sleep(5)
html=driver.page_source
soup=BeautifulSoup(html,'html.parser')
div=soup.select_one("div.sr-table-content")
print(div)

Import

from selenium.webdriver.common.by import By
from time import sleep
user_99012610

You can extract tables from webpages using read_html() function from pandas library. It can Effectively Scrape the Webpage and extract the Table from the page. You can Even Manupilate values or Directly save it to a csv file.

Get more information about pandas here or read more about that specific function here

Hope it helps your usecase.

user_154227310

Try this code once Just edit the class_names and other urls to get the table

# import libraries
import urllib2
from bs4 import BeautifulSoup
# specify the url
quote_page = ‘http://www.bloomberg.com/quote/SPX:IND'
# query the website and return the html to the variable ‘page’
page = urllib2.urlopen(quote_page)
# parse the html using beautiful soup and store in variable `soup`
soup = BeautifulSoup(page, ‘html.parser’)
# Take out the <div> of name and get its value
name_box = soup.find(‘h1’, attrs={‘class’: ‘name’})
name = name_box.text.strip() # strip() is used to remove starting and trailing
print name

I'm Leaving some screen shots here please refer it worked for me

Image1 Image2

user_154337870
Related