Selenium can't find divs when deploying in Heroku

Viewed 92

I am trying to grab the 2 tables from this link: https://www.nba.com/standings?GroupBy=conf&Season=2019-20&Section=overall

I can grab them like this when my app is not deployed to Heroku

import requests
import os
from bs4 import BeautifulSoup
from selenium import webdriver

url = 'https://www.nba.com/standings?GroupBy=conf&Season=2019-20&Section=overall'
driver = webdriver.Chrome()
driver.get(url)

soup = BeautifulSoup(driver.page_source, 'html.parser')
tables = soup.select('div.StandingsGridRender_standingsContainer__2EwPy')

but when I deploy my app to Heroku, I have to specify the chromedriver and this is what my code looks like:

url = 'https://www.nba.com/standings?GroupBy=conf&Season=2019-20&Section=overall'
CHROMEDRIVER_PATH = "/app/.chromedriver/bin/chromedriver"

chrome_bin = os.environ.get('GOOGLE_CHROME_BIN', "chromedriver")
options = webdriver.ChromeOptions()
options.binary_location = chrome_bin
options.add_argument("--disable-gpu")
options.add_argument("--no-sandbox")
options.add_argument('headless')
options.add_argument('window-size=1200x600')
options.add_argument('--disable-dev-shm-usage') 
options.add_argument("--disable-popup-blocking")
driver = webdriver.Chrome(executable_path=CHROMEDRIVER_PATH, chrome_options=options)
driver.get(url)

soup = BeautifulSoup(driver.page_source, 'html.parser')
tables = soup.select('div.StandingsGridRender_standingsContainer__2EwPy')

Whenever I run this code on Heroku I get an IndexError saying that the list index is out of range (I pull the tables from tables[0] and tables[1] later on in my code)

Is there a way I can figure out why the tables are not getting grabbed? Is there a pop up that is keeping the driver from accessing the tables? I tried to specify --disable-popup-blocking but have had no luck. Thanks!

1 Answers

If you want to scrape the table then why are you using bs4?

You can use only selenium and use"find_element_by_xpath", "find_element_by_tage_name", "find_element_by_class_name"...etc

when you got the value then use ".text" to extract the text.

Related