How do I parse from BeautifuSoup Comments

user_13805703 0
Viewed 53

I am doing some data mining on football stats using Python's BeautifulSoup. There are some tables that give issues when trying to filter through the soup. Upon further inspection it looks like the data I need is contained within a comment, which is not the case when looking at it through the web developer tools.

import requests
from bs4 import BeautifulSoup, Comment

url='https://aws.pro-football-reference.com/teams/mia/2000.htm'
page = requests(url)
soup = BeautifulSoup(page, 'html.parser')
table = soup.find(id='all_passing')
print(table)

Below is a sample of what is printed.

<div class="table" id="all_passing"> <div class="placeholder"></div> <!--    <div class="table_outer_container">
      <div class="overthrow table_container" id="div_passing">
         <table class="sortable stats_table" id="passing" <caption>Passing Table</caption>    <colgroup><col><col><col></colgroup>    <thead>      
      <tr>
         <th aria-label="Uniform number" data-stat="uniform_number" scope="col">No.</th>
         <th aria-label="Player's age" data-stat="age" scope="col">Age</th>          <th aria-label="Position" data-stat="pos" scope="col">Pos</th>
      </tr>
          </thead>    <tbody> <tr ><th scope="row" class="right " data-stat="uniform_number" >9</th><td class="right " data-stat="age"
>29</td><td class="left " data-stat="pos" >QB</td></tr>    </tbody> </table>

      </div>    </div>
--> <div class="placeholder"></div> </div>

How do I go about filtering down the comments? This is what I have tried.

comments = table.find_all(text=lambda text:isinstance(text, Comment))
rows = comments[0].find_all('tr')
print('rows: ' + rows)

This prints:

None
beautifulsoup python-3.x python
1 Answers

You can iterate .contents of the tag and check if the type of content is Comment. The you could load the comment into the new soup:

import requests
from bs4 import BeautifulSoup, Comment


url = 'https://aws.pro-football-reference.com/teams/mia/2000.htm'
soup = BeautifulSoup(requests.get(url).content, 'html.parser')

# find all passing table:
for c in soup.select_one('#all_passing').contents:
    if isinstance(c, Comment):
        break
all_passing = BeautifulSoup(c, 'html.parser')

# print some data to screen:
for tr in all_passing.select('tr'):
    print(tr.get_text(strip=True, separator='\t'))

Prints:

No. Player  Age Pos G   GS  QBrec   Cmp Att Cmp%    Yds TD  TD% Int Int%    Lng Y/A AY/A    Y/C Y/G Rate    QBR Sk  Yds NY/A    ANY/A   Sk%4QC  GWD
9   Jay Fiedler 29  QB  15  15  10-5-0  204 357 57.1    2402    14  3.9 14  3.9 61  6.7 5.7 11.8    160.1   74.5    23  129 5.98    5.06    6.11    1
11  Damon Huard 27  qb  16  1   1-0-0   39  63  61.9    318 1   1.6 3   4.8 29  5.0 3.2 8.2 19.9    60.2    4   22  4.42    2.70    6.01    1
26  Lamar Smith 30  RB  15  15  0   1   0.0 0   0   0.0 0   0.0 0   0.0 0.0 0.0 39.6    0   0   0.00    0.00    0.0
34  Thurman Thomas  34  9   0   0   0   0   0   0   0   0.0 1   2   -2.00   -2.00   100.0
Team Total  27.3    16  11-5-0  243 421 57.7    2720    15  3.6 17  4.0 61  6.5 5.4 11.2    170.0   72.2    28  153 5.72    4.68    6.2 2   2
Opp Total   16  282 530 53.2    3170    13  2.5 28  5.3 6.0 4.09    11.2    198.1   57.5    48  270 5.0 3.3 8.3
user_100359850
Related