How to extract contents of a table from webpage using Python?

user_14036987 0
Viewed 195

I need help in extracting kmz and zip files from a webpage. The following code is able to extract the table, but not the files and links inside the table. What can I include in my code so that the output table also contains the links and files instead of just plain text?

Webpage:

https://www.nhc.noaa.gov/gis/

Code:

import pandas as pd
url = 'https://www.nhc.noaa.gov/gis/'
result = pd.read_html(url)[0]
result
pandas web-scraping beautifulsoup python-3.x python
2 Answers

I'd say use beautifulsoup (bs4) instead of pandas to parse html.

pip install beautifulsoup4 requests

and then it's as simple as

import bs4
import requests

result = bs4.BeautifulSoup(requests.get('https://www.nhc.noaa.gov/gis/').content, features='html.parser')
for link in result.find('table').find_all('a'):
    print(link.attrs['href'])
user_140285360

You can use beautifulsoup to get all the links.

from bs4 import BeautifulSoup
import requests
import pandas as pd
url = 'https://www.nhc.noaa.gov/gis/'

res = requests.get(url)
soup = BeautifulSoup(res.text, "html.parser")
table = soup.find("table")
for anchor in table.find_all("a"):
    print("Text - {}, Link - {}".format(anchor.get_text(strip=True), anchor["href"]))

Output:

Text - Irma Example, Link - /gis/examples/al112017_5day_020.zip
Text - Cone, Link - /gis/examples/AL112017_020adv_CONE.kmz
Text - Track, Link - /gis/examples/AL112017_020adv_TRACK.kmz
Text - Warnings, Link - /gis/examples/AL112017_020adv_WW.kmz
Text - shp, Link - forecast/archive/al092020_5day_latest.zip
Text - Cone, Link - /storm_graphics/api/AL092020_CONE_latest.kmz
Text - Track, Link - /storm_graphics/api/AL092020_TRACK_latest.kmz
Text - Warnings, Link - /storm_graphics/api/AL092020_WW_latest.kmz

If you want to retain the dataframe, don't make a network call again via read_html. Reuse the response object.

df = pd.read_html(res.text)

To get the complete link, append the following to all the links.

https://www.nhc.noaa.gov

Code:

for anchor in table.find_all("a"):
    print("Text - {}, Link - {}".format(anchor.get_text(strip=True), prefix + anchor["href"]))

Output:

Text - Irma Example, Link - https://www.nhc.noaa.gov/gis/examples/al112017_5day_020.zip
Text - Cone, Link - https://www.nhc.noaa.gov/gis/examples/AL112017_020adv_CONE.kmz
Text - Track, Link - https://www.nhc.noaa.gov/gis/examples/AL112017_020adv_TRACK.kmz
Text - Warnings, Link - https://www.nhc.noaa.gov/gis/examples/AL112017_020adv_WW.kmz
Text - shp, Link - https://www.nhc.noaa.govforecast/archive/al092020_5day_latest.zip
Text - Cone, Link - https://www.nhc.noaa.gov/storm_graphics/api/AL092020_CONE_latest.kmz
Text - Track, Link - https://www.nhc.noaa.gov/storm_graphics/api/AL092020_TRACK_latest.kmz
Text - Warnings, Link - https://www.nhc.noaa.gov/storm_graphics/api/AL092020_WW_latest.kmz

To download the files, again use requests and download the file

user_68496820
Related