Scrape table content with dropdown list with BeautifulSoup

user_3908237 0
Viewed 155

I'd like to scrape ALL search results from https://www.cbssports.com/nfl/playersearch?POSITION=RB&print_rows=9999 for all the players of ALL positions. enter image description here

I have achieved to get all RB players with the following code:

from bs4 import BeautifulSoup
import requests

html_text = requests.get('https://www.cbssports.com/nfl/playersearch?POSITION=RB&print_rows=9999')
html = html_text.text
soup = BeautifulSoup(html, 'html.parser')
player_table = soup.find('table', class_='data')

for tr in all_player_table.find_all('tr', class_=['row1','row2']):
    tds = tr.find_all('td')
    print(("Player:%s , Position:%s , Team: %s") % (tds[0].text, tds[1].text, tds[2].text))

I am now facing to deal with scraping players in other positions from the dropdown list. What is the best way to do it?

web-scraping beautifulsoup python
1 Answers

The idea is pretty simple: You can scrape all the positions, modify the URL and search for all players. In code:

from bs4 import BeautifulSoup
import requests

main_url = "https://www.cbssports.com/nfl/playersearch"
soup = BeautifulSoup(requests.get(main_url).text, "html.parser")

# Scrape all positions
positions = [o["value"] for o in soup.find("select", {'name' : "POSITION"}).find_all("option")]

for position in positions:
    url = f"{main_url}?POSITION={position}&print_rows=9999"
    # Find all players
    soup = BeautifulSoup(requests.get(url).text, "html.parser")
    for tr in soup.find("table", class_="data").find_all("tr", class_=["row1", "row2"]):
        tds = tr.find_all('td')
        print(("Player: %s , Position: %s , Team: %s") % (tds[0].text, tds[1].text, tds[2].text))
user_47455290
Related