I am scraping NFL passing data for years 1971 to 2019. I was able to scrape the data on the first page of each year using this code:
# This code works:
passingData = [] # create empty list to store column data
for year in range(1971,2020):
url = 'https://www.nfl.com/stats/player-stats/category/passing/%s/REG/all/passingyards/desc' % (year)
response = requests.get(url)
response = response.content
parsed_html = bsoup(response, 'html.parser')
data_rows = parsed_html.find_all('tr')
passingData.append([[col.text.strip() for col in row.find_all('td')] for row in data_rows])
The first page for each year only has 25 players, and roughly 70-90 players threw a pass each year (so there are between 3-4 pages of player data on "subpages" within each year). The problem comes when I try to scrape these subpages. I tried to add another sub-for-loop which pulls out the href of each link to the next page and append to the base url which is found in the div class 'nfl-o-table-pagination__buttons'
Unfortunately, I cannot add to the passingData list from the first page. I attempted the below, but 'Index Out of Range Error' occurred on the subUrl line.
I am still new to web scraping, so if my logic is way off please let me know. I figured I could just append the subpage data (since the table structure is the same), but seems the error arises when I attempt to go from:
https://www.nfl.com/stats/player-stats/category/passing/%s/REG/all/passingyards/desc
to the second page, which has url of :
https://www.nfl.com/stats/player-stats/category/passing/2019/REG/all/passingYards/DESC?aftercursor=0000001900000000008500100079000840a7a000000000006e00000005000000045f74626c00000010706572736f6e5f7465616d5f737461740000000565736249640000000944415234363631343100000004726f6c6500000003504c5900000008736561736f6e496400000004323031390000000a736561736f6e5479706500000003524547f07fffffe6f07fffffe6389bd3f93412939a78c1e6950d620d060004
for subPage in range(1971,2020):
subPassingData = []
subUrl = soup.select('.nfl-o-table-pagination__buttons a')[0]['href']
new = requests.get(f"{url}{subUrl}")
newResponse = new.content
soup1 = bsoup(new.text, 'html.parser')
sub_data_rows = soup1.find_all('tr')
subPassingData.append([[col.text.strip() for col in row.find_all('td')] for row in data_rows])
passingData.append(subPassingData)
Thank you for your help.