Can't get Table Data from Site After the Page Loads

user_17171923 0
Viewed 55

I'm trying to obtain the 2nd table on the following website. I've tried BS4, Pandas, and now selenium, but I can't obtain the table for the life of me.

The table data doesn't load until after the page comes up.

There is a dictionary on the 'view source' page with the info, but its seems like every element on that page is 'line-content', so its makes it difficult to obtain only the desired table info.

What is the best way to collect the table data?

from ast import Return
from selenium import webdriver
from selenium.webdriver.common.keys import Keys
from selenium.webdriver.support.ui import WebDriverWait
from selenium.webdriver.common.by import By
from selenium.webdriver.support import expected_conditions as ec
import time

path = 'C:\Program Files (x86)\chromedriver.exe'

driver = webdriver.Chrome(path)

driver.get('https://www.linestarapp.com/Ownership/Sport/NFL/Site/DraftKings/PID/295')

time.sleep(5)
driver.maximize_window()

time.sleep(5)

players = driver.find_elements_by_class_name('playerRow').text
print(players)
pandas selenium beautifulsoup python
3 Answers

There are several issues here:

  1. find_elements_by_class_name method returns a list of web elements. You can not apply .text method on a list of web elements, you need to iterate over elements in a list and extract text from each element one by one.
  2. You should not use hardcoded pauses like time.sleep(5) Expected Conditions explicitly waits should be used instead.
    Something like this:
from ast import Return
from selenium import webdriver
from selenium.webdriver.common.keys import Keys
from selenium.webdriver.support.ui import WebDriverWait
from selenium.webdriver.common.by import By
from selenium.webdriver.support import expected_conditions as EC
import time

path = 'C:\Program Files (x86)\chromedriver.exe'

driver = webdriver.Chrome(path)
wait = WebDriverWait(driver, 20)

driver.get('https://www.linestarapp.com/Ownership/Sport/NFL/Site/DraftKings/PID/295')

driver.maximize_window()

wait.until(EC.visibility_of_element_located((By.CSS_SELECTOR, ".playerRow")))
#short pause added in order to make sure all the elements are loaded after we know the first element was loaded
time.sleep(0.5)

players = driver.find_elements_by_class_name('playerRow')
for player in players:
    print(player.text)

UPD
Using this css_selector locator #tableTournament .playerRow instead of just .playerRowwill give you the right table rows

user_34854340

Another solution to your problem might be to query for the second table by id, and only later get the table row elements.

from selenium import webdriver
    
driver = webdriver.Chrome("C:\Program Files (x86)\chromedriver.exe")
 
driver.get('https://www.linestarapp.com/Ownership/Sport/NFL/Site/DraftKings/PID/295')

driver.maximize_window()

# the first table has id projectedTournament
secondTable = driver.find_element_by_id("tableTournament")

players = secondTable.find_elements_by_class_name('playerRow')
for playerRowElement in players:
    for playerColumn in playerRowElement.find_elements_by_tag_name("td"):
        print(playerColumn.text)
user_57587380

Just an alternative to Selenium, the table is in the html actually in the <script> tag.

import requests
import pandas as pd
import re
import json

url = 'https://www.linestarapp.com/Ownership/Sport/NFL/Site/DraftKings/PID/295'

response = requests.get(url)

p = 'var actualResultsDict = ({*.})'

replaceDict = {"'":'"', 
               'id': '"id"',
               'name':'"name"',
               'owned':'"owned"',
               'pos':'"pos"',
               'team':'"team"',
               'sal':'"sal"'}

jsonStr = re.search(r'var actualResultsDict = {([\w\W]*)function posMatch', response.text, re.M).group(1)

for key, value in replaceDict.items():
    jsonStr = jsonStr.replace(key, value)
    
jsonStrSplit = jsonStr.split('],')    

jsonData = {}
for idx, each in enumerate(jsonStrSplit[:-1]):
    temp = '{' + each.rsplit(',',1)[0] + ']}'
    tempData = json.loads(temp)
    
    jsonData.update(tempData)
    
    
rows = []
for k,v in jsonData.items():
    rows += v

df = pd.DataFrame(rows)

Output:

print(df)
       id                   name  owned pos team   sal
0    5942       Devin Singletary  37.31  RB  BUF  5900
1    7303      Ja&#39;Marr Chase  33.57  WR  CIN  7100
2    7658           Eli Mitchell  28.31  RB   SF  5800
3    3612          Derrick Henry  26.93  RB  TEN  7500
4    2142          Davante Adams  26.25  WR   GB  8500
..    ...                    ...    ...  ..  ...   ...
108  6895  Nick Westbrook-Ikhine   1.72  WR  TEN  3800
109  6006          Scotty Miller   1.70  WR   TB  3300
110  4453     D&#39;Onta Foreman   1.64  RB  TEN  5200
111  2403            Geoff Swaim   1.36  TE  TEN  2800
112  5359      Dontrell Hilliard   1.08  RB  TEN  4300

[113 rows x 6 columns]
user_77522960
Related