I'm parsing data from this website. I've parsed several things from the site, but for some reason, this particular page isn't finding one of the tables. Here's a simplified code snippet which demonstrates the issue:
#!/usr/bin/env python3
import bs4 as bs
import requests
def get_soup(site):
headers = {'User-Agent': 'Mozilla/5.0'}
r = requests.get(site, headers=headers)
#Always want a status code of 200, which means everything downloaded
if r.status_code != 200:
print(r.status_code)
print("Invalid Status Code")
exit(1)
return bs.BeautifulSoup(r.content, 'html.parser')
soup = get_soup("https://www.hockey-reference.com/boxscores/202008140WSH.html#all_advanced")
table = soup.find('table' , {'id' : "NYI_skaters"}).find('tbody').find_all('tr')
table = soup.find('table' , {'id' : "NYI_goalies"}).find('tbody').find_all('tr')
table = soup.find('table' , {'id' : "NYI_adv"}).find('tbody').find_all('tr')
The code finds the skaters and goalies table with no issues, but it doesn't find the _adv table, resulting in a NoneType error, since it's not finding the table. I'm able to find a node which _adv resides under:
table = soup.find('div' , {'id' : "all_advanced"})
Underneath that div tag (all_advanced), there's some strange looking code, so I'm not sure if that has something to do with it. I haven't had any issues with this particular site with other things, never needed to use selenium and they're fine with people scraping the data. Any help would be appreciated.
EDIT: Using panda.read_html doesn't find it either. I was able to work around it by replacing all the "table =" above with:
for comment in soup.find_all(text=lambda text: isinstance(text, bs.Comment)):
if comment.find("<table ") > 0:
comment_soup = bs.BeautifulSoup(comment, 'html.parser')
table = comment_soup.find('table' , {'id' : "NYI_adv"})
for player in table.find_all('tr' , {'class' : "ALLSH hidden"}):
print(player.find('a')['href'])
Thanks