How to scrape a specific tag without class name?

Viewed 31

I'm developing a python script to scrape data from a specific site:
https://www.abs.gov.au/statistics/economy/price-indexes-and-inflation/residential-property-price-indexes-eight-capital-cities/latest-release

I'm using BeautifulSoup. The interesting data on the HTML page are in this structure:

<div class="table-scrolling-wrapper">
                  <div class="abs-content clearfix text-formatted field field--name-field-abs-text-paragraph-content field--type-text-long field--label-hidden">
                   <table>
                    <thead>
                     <tr>
                      <th colspan="1" rowspan="2">
                       Residential property prices
                      </th>
                      <th class="text-align-right">
                       Sep Qtr 21 to Dec Qtr 21
                      </th>
                      <th class="text-align-right">
                       Dec Qtr 20 to Dec Qtr 21
                      </th>
                     </tr>
                     <tr>
                      <th class="text-align-right">
                       % change
                      </th>
                      <th class="text-align-right">
                       % change
                      </th>
                     </tr>
                    </thead>
                    <tbody>
                     <tr>
                      <td>
                       Sydney
                      </td>
                      <td class="text-align-right">
                       4.1
                      </td>
                      <td class="text-align-right">
                       26.7
</td>

I would like to take Sydney, 4.1, 26.7

I have tried this way:

    from bs4 import BeautifulSoup
    import requests
    
    link='https://www.abs.gov.au/statistics/economy/price-indexes-and-inflation/residential-property-price-indexes-eight-capital-cities/latest-release'
    
    url=requests.get(link).text
soup=BeautifulSoup(url,'lxml')
table=soup.find_all('div',class_='abs-content clearfix text-formatted field field--name-field-abs-text-paragraph-content field--type-text-long field--label-hidden')
for data in table:
    region=data.find_all('tr')
    for header in region:
        figures=header.find_all('td',class_='text-align-right')
        names=header.find('td')
        for figure,name in zip(figures,names):
            print(figure.text)
            print(name.text)

Now I am able to get 4.1 & 26.7. But I am not able to get Sydney. I can only get this <td>Sydney</td>

Once I tried to print(name.text), Error occur

'NoneType' object has no attribute 'text'

How can I ignore the error, and only get those value which are not None? Thank you.

1 Answers

Alexander has already given you correct answer:

that is because sydney td doesn't have a class so just remove the class_=text-align-right

It will return you a list consisting of three elements, where Sydney will be the first, and the others will be second and third respectively.

Related