Parsing data with beautiful soup, targeting data- attribute

user_3131132 0
Viewed 52

A website has the following html elements:

<td class="right " data-stat="week_num">1</td>
<td class="right " data-stat="week_num">2</td>
<!-- etc -->

I was able to grab those elements with the following code:

import requests
from bs4 import BeautifulSoup
    
url = "https://www.pro-football-reference.com/players/H/HopkDe00.htm"
page = requests.get(url)
soup = BeautifulSoup(page.text, 'html.parser')

for item in soup.find_all(attrs={'data-stat':'week_num'}):
  print(item)

This gets the requested html elements without an issue.

There is another set of elements I am trying to retrieve:

<tr id = 'stats.111' data-row='0'>
<tr id = 'stats.112' data-row='1'>
<!-- etc -->

To get these I figured I only had to alter the above code slightly. But this did not work. Note below that I tried passing 'True' as a string and just True, the program ran both times but no elements were printed to the console.

for item in soup.find_all(attrs={'data-row':True}): # attempted to get all elements with the `data-row` attribute, this returned `None`. 
    print(item)

I then attempted to just get a single element to test if I could do that with the same code.

for item in soup.find_all(attrs={'data-row':'1'}): # target just the <tr data-row='1'> element  
    print(item)

but this also returned nothing. How can I target this group of elements with the data-row attribute?

beautifulsoup
1 Answers

The data-row attribute is added dynamically by JavaScript, so the rows need to be targeted differently. For example get all rows under the table with id="stats":

import requests
from bs4 import BeautifulSoup


url = 'https://www.pro-football-reference.com/players/H/HopkDe00.htm'
soup = BeautifulSoup(requests.get(url).content, 'html.parser')

for row in soup.select('table#stats tbody tr'):
    tds = [td.get_text(strip=True) for td in row.select('td, th')]
    print(*tds)

Prints:

2020-09-13 1 ARI @ SFO W 24-20 * 16 14 151 10.79 0 87.5% 9.44 0 0 0 0 0 0 0 77 94% 0 0% 0 0%
2020-09-20 2 ARI  WAS W 30-15 * 9 8 68 8.50 1 88.9% 7.56 1 0 0 0 0 0 0 75 97% 0 0% 0 0%
2020-09-27 3 ARI  DET L 23-26 * 12 10 137 13.70 0 83.3% 11.42 0 0 0 0 0 0 0 61 94% 0 0% 0 0%
2020-10-04 4 ARI @ CAR L 21-31 * 9 7 41 5.86 0 77.8% 4.56 0 0 0 0 0 0 0 54 95% 0 0% 0 0%

...and so on.
user_100359850
Related