I'm developing a python script to scrape data from a specific site:
https://www.abs.gov.au/statistics/economy/price-indexes-and-inflation/residential-property-price-indexes-eight-capital-cities/latest-release
I'm using BeautifulSoup. The interesting data on the HTML page are in this structure:
<div class="table-scrolling-wrapper">
<div class="abs-content clearfix text-formatted field field--name-field-abs-text-paragraph-content field--type-text-long field--label-hidden">
<table>
<thead>
<tr>
<th colspan="1" rowspan="2">
Residential property prices
</th>
<th class="text-align-right">
Sep Qtr 21 to Dec Qtr 21
</th>
<th class="text-align-right">
Dec Qtr 20 to Dec Qtr 21
</th>
</tr>
<tr>
<th class="text-align-right">
% change
</th>
<th class="text-align-right">
% change
</th>
</tr>
</thead>
<tbody>
<tr>
<td>
Sydney
</td>
<td class="text-align-right">
4.1
</td>
<td class="text-align-right">
26.7
</td>
I would like to take Sydney, 4.1, 26.7
I have tried this way:
from bs4 import BeautifulSoup
import requests
link='https://www.abs.gov.au/statistics/economy/price-indexes-and-inflation/residential-property-price-indexes-eight-capital-cities/latest-release'
url=requests.get(link).text
soup=BeautifulSoup(url,'lxml')
table=soup.find_all('div',class_='abs-content clearfix text-formatted field field--name-field-abs-text-paragraph-content field--type-text-long field--label-hidden')
for data in table:
region=data.find_all('tr')
for header in region:
figures=header.find_all('td',class_='text-align-right')
names=header.find('td')
for figure,name in zip(figures,names):
print(figure.text)
print(name.text)
Now I am able to get 4.1 & 26.7. But I am not able to get Sydney. I can only get this <td>Sydney</td>
Once I tried to print(name.text), Error occur
'NoneType' object has no attribute 'text'
How can I ignore the error, and only get those value which are not None? Thank you.