Something wrong on Web Scraping using BeautifulSoup

user_10707440 0
Viewed 181

Trying to extract the list of teams in the table of the Brazilian Football Championship, searching for the correct tag and class, I didn't get any returning data. I have tried to read the official documentation on the BS4 website but still I am still unable to solve this problem. I will be grateful if someone can help me. Below are the screenshots and code used.

The page: https://i.stack.imgur.com/9YTJf.png

Using the element inspector by a selector: https://i.stack.imgur.com/asMng.png

Developer Tool Window: https://i.stack.imgur.com/fAKZJ.png

The search with no data returned: https://i.stack.imgur.com/U2S5W.png

from bs4 import BeautifulSoup
import lxml, requests

r = requests.get('https://www.google.com/search?q=Tabela+do+Campeonato+Brasileiro+de+Futebol&oq=Tabela+do+Campeonato+Brasileiro+de+Futebol&aqs=chrome..69i57.241j0j1&sourceid=chrome&ie=UTF-8#sie=lg;/g/11fmzksb3y;2;/m/0fnk7q;st;fp;1;;')

page = r.text
soup = BeautifulSoup(page, 'lxml')

for i in soup.find_all('span', class_='ellipsisize hsKSJe'):
    print(i.text)
web-scraping beautifulsoup python
1 Answers

I believe this issue is because you are trying to use beautiful soup to get data from a dynamic page. For that you can use selenium with the chrome driver. I save it in its own folder (bin\chromedriver.exe) on my the system drive.

The below gets you the first five rows for instance (didn't have enough persistence to figure out the selector for everything else, sorry!)

from selenium import webdriver
import pandas as pd

URL = 'https://www.google.com/search?q=Tabela+do+Campeonato+Brasileiro+de+Futebol&oq=Tabela+do+Campeonato+Brasileiro+de+Futebol&aqs=chrome..69i57.241j0j1&sourceid=chrome&ie=UTF-8#sie=lg;/g/11fmzksb3y;2;/m/0fnk7q;st;fp;1;;'
#webdriver and get data from dynamic page
dr = webdriver.Chrome(executable_path=r'C:/bin/chromedriver.exe')
dr.get(URL)
#get table data by xpath
data = dr.find_element_by_css_selector('#rso').get_attribute('outerHTML')
dr.close()

#get data as dataframe
raw = pd.read_html(data)[0]
#organize retrieved columns
labels = raw.columns.values[1:11]
table = raw[labels]
#delete excess column
del table['Club']
table.columns = labels[:-1] #ignore the last value

#view table (can't post an image yet! new here :))
user_143030950
Related