Python BeautifulSoup html parsing

user_9560888 0
Viewed 65

New to Webscraping, and trying to parse html of http://www.airportcodes.org and get airport codes along with city name and state into a dataframe. HTML looks like the below. Basically it seems like after every 'div class="t6" tag, lines of items (city, state, and airport code) I need to parse are listed there with a break tag, but since they are sectioned alphabetically, the number of items in each section differs.

<table border="0" cellpadding="0" cellspacing="0" width="100%">
<tr><td bgcolor="#CBC7E0"><img height="2" src="img/0.gif" width="1"/></td></tr>
</table></div>
<a name="USA"></a>
<div class="t6"><div class="major"><b>A</b></div></div>

Aberdeen, SD (ABR) <br/>
Abilene, TX (ABI)<br/>
Adak Island, AK (ADK)<br/>
Akiachak, AK (KKI)<br/>
Akiak, AK (AKI)<br/>
Akron/Canton, OH (CAK)<br/>
Akuton, AK (KQA)<br/>
Alakanuk, AK (AUK)<br/>
Alamogordo, NM (ALM)<br/>
...

<div class="sectionBreak">
<table border="0" cellpadding="0" cellspacing="0" width="100%">
<tr><td bgcolor="#CBC7E0"><img height="2" src="img/0.gif" width="1"/></td></tr>
</table></div>
<div align="right"><div align="right"><a href="#top">return to top</a></div></div>
<a name="USB"></a>
<div class="t6"><div class="major"><b>B</b></div></div>

Bakersfield, CA (BFL)<br/>
Baltimore, MD (BWI)<br/>
Bangor, ME (BGR)<br/>
Bar Harbour, ME (BHB)<br/>
...

Codes so far:

import bs4 as bs
sauce = urllib.request.urlopen('http://www.airportcodes.org/').read()
soup = bs.BeautifulSoup(sauce, 'lxml')

for tag in soup.find_all("div", class_="t6"):
    print(tag.next_sibling)

gets me:

Aberdeen, SD (ABR) 


Bakersfield, CA (BFL)


Canton/Akron, OH (CAK)

But this is wrong as it looks like the code is only returning the first item next to the specified tag. Any help would be appreciated!

beautifulsoup css parsing html python
2 Answers

First, you have to extract the br tags, to read properly the text result.

Then you can loop over type(tag) and test if it is a bs4.element.NavigableString

Try this:

import requests
from bs4 import BeautifulSoup as soup
import bs4

url = "http://www.airportcodes.org/"
resp = requests.get(url)

page = soup(resp.text, 'html.parser')
for br in page.find_all('br'):
    br.extract()

for tag in page.find_all("div", class_="t6"):
    while type(tag.next_sibling) is bs4.element.NavigableString:
        tag = tag.next_sibling
        print(tag)

Result:

Aberdeen, SD (ABR) 
Easton, PA (ABE)
...
...
Island Lake/Garden Hill, Canada (YIV)
Islay, United Kingdom (ILY)
user_24371410

Use find_next_siblings(text=True) to get all the Airport Info.

from bs4 import BeautifulSoup
import requests
r=requests.get("http://www.airportcodes.org/").content
soup=BeautifulSoup(r,'html.parser')
for items in soup.select('div.t6'):
   for item in items.find_next_siblings(text=True):
       print(item)
user_108856840
Related