Extracting Hyperlinks from Basketball Reference Column(On pages With multiple Tables) to new Column

idekman 0
Viewed 26

I have recently been working on a scraper for NBA MVPS from basketball reference and hope to incorporate the embedded Player page hyperlinks that appear into a new column in the same row as the player. I have done the same for scraping other pages but unfortunately, due to the many tables and indeterminate order of them, my prior method returns many random links from around the page. Below is my code, the column player links is the one in question. While these links are acceptably formatted, they are simply not the correct ones as stated prior. The table is perfectly fine however this column is the problem.

import requests
from bs4 import BeautifulSoup
import pandas as pd 
url='https://www.basketball-reference.com/awards/awards_2022.html'
html = requests.get(url).text.replace('<!--', '').replace('-->', '')
soup = BeautifulSoup(html, "html.parser")
tabs = soup.select('table[id*="mvp"]')
for tab in tabs:
    cols, players = [], []
    for s in tab.select('thead tr:nth-child(2) th'):
        cols.append(s.text)
    for j in (tab.select('tbody tr, tfoot tr')):
        player = [dat.text for dat in j.select('td,th') ]
        players.append(player) 
links= []
for link in soup.findAll('table')[1].findAll('a'):
    url = link.get('href')
    links.append(url)   
my_list = links
substring = '/friv/'
new_list = [item for item in my_list if not item.startswith(substring)]
for item in my_list.copy():
    if substring in item:
        my_list.remove(item)
players_plus = [player + [""]*(max_length - len(player)) for player in players]
df=pd.DataFrame(players_plus,columns=cols)

df['Playerlinks']=my_list


print(df.to_markdown())

So, essentially I am asking whether anyone is aware of a method to scrape just these hyperlinks(12 in the given example), and put them either into an ordered list(to be put into a column) or any better methods you may be aware of. My expected output, for this link, in particular, would be a first-row value of "/players/j/jokicni01.html", a second of "/players/e/embiijo01.html" etc; corresponding with their respective players. I have tried many methods using ids, find alls, and others but unfortunately my Html knowledge is simply very limited and I am starting to go in circles. Thank you in advance for any help you can provide.

beautifulsoup web-scraping pandas python
1 Answers

Does something like this work for you? You can extract the profile links while you are iterating through the player rows.

for tab in tabs:
    cols, players = [], []
    for s in tab.select('thead tr:nth-child(2) th'):
        cols.append(s.text)
    for j in (tab.select('tbody tr, tfoot tr')):
        player = [dat.text for dat in j.select('td,th') ]

        # get player link
        player.append(j.find('a')['href'])
      
        players.append(player) 

df=pd.DataFrame(players,columns=cols+["Playerlinks"])
print(df)
user3iV70
Related