I would like to download automatically the table from this link: https://www.barchart.com/options/iv-rank-percentile/stocks
In order to do this, with the help of some tutorials, I wrote this piece of code:
# Import libraries
from urllib.request import Request, urlopen
import requests
from bs4 import BeautifulSoup as soup
# Set the URL you want to webscrape from
url = 'https://www.barchart.com/options/iv-rank-percentile/stocks?viewName=main'
# Connect to the URL
response = requests.get(url)
print(response)
req = Request(url, headers={'User-Agent': 'Mozilla/5.0'})
webpage = urlopen(req).read()
print(req)
# Parse HTML and save to BeautifulSoup object¶
page_soup = soup(webpage, "html.parser")
#print(page_soup)
containers = page_soup.findAll("a", "toolbar-button download")
for container in containers:
print(container)
url = container.get('href')
print(url)
The results I printed are the followings:
<Response [403]>
<urllib.request.Request object at 0x030766F0>
<a class="toolbar-button download" data-bc-download-button=" Stocks IV Rank and IV Percentile "> <i class="bc-glyph-download"></i> <span>download</span></a>
None
It seems that I'm not able to find the 'href'
At this point, I'm struggling a bit with the next steps, because I don't really know how I can download the file (since the 'href' can't be found).
Maybe someone can help / or propose another solution?
Thanks a lot in advance,
Marketwizard