Webscraping: Downloading a pdf from a javascript link

user_10202292 0
Viewed 825

I am using the requests library in python and attempting to scrape a website that has lots of public reports and documents in .pdf format. I have successfully done this on other websites, but I have hit a snag on this one: the links are javascript functions (objects? I don't know anything about javascript) that redirect me to another page, which then has the raw pdf link. Something like this:

import requests
from bs4 import BeautifulSoup as bs

url = 'page with search results.com'
html = requests.get(url).text
soup = bs(html)
obj_list = soup.findAll('a')
for a in obj_list: 
    link = a['href']
    print(link)

>> javascript:readfile2("F","2201","2017_2201_20170622F14.pdf")

Ideally I would like a way to find what url this would navigate to. I could use selenium and click on the links, but there are a lot of documents and that would be time- and resource-intensive. Is there a way to do this with requests or a similar library?

Edit: It looks like every link goes to the same url, which loads a different pdf depending on which link you click. This makes me think that there is no way to do this in requests, but I am still holding out hope for something non-selenium-based.

python-requests web-scraping python
1 Answers

There might be a default url on which these PDF files are present.

You need to find out the url, On which these pdf files open after clicking on hyper link.

Once you got that url, You need to parse pdf name from anchor text.

Afterwards, You append the pdf name with url(On which pdf is present). And request the final url.

user_33955370
Related