Extract Weblinks from multiple Web

Viewed 91

I need to extract weblinks to download PDF files from these pages, I am thinking of extracting the weblinks from the website and then filter out the PDF links and use a download manager to download all pdf files.. how can I use multiple links in the python code to download all the links, it's working for only one weblink.

from bs4 import BeautifulSoup    
urls = 'https://www.nzx.com/announcements/190008'    
urls = 'https://www.nzx.com/announcements/372918'     
urls = 'https://www.nzx.com/announcements/372980'            

urls = 'https://www.nzx.com/announcements/373053'

grab = requests.get(urls)
soup = BeautifulSoup(grab.text, 'html.parser')
# opening a file in write mode
f = open("t.txt", "w")
# traverse paragraphs from soup
for link in soup.find_all("a"):

data = link.get('href')

f.write(data)

f.write("\n")

f.close()***```
2 Answers

if you want to do it quick and dirty so to say do it like @Sharku sugessted and make a list

BASE_URL = 'https://www.nzx.com/announcements/'

pdf_list = ['190008', '372918', '372980']

for pdf in pdf_list:
    grab = requests.get(f'{BASE_URL}{pdf}')
    soup = BeautifulSoup(grab.text, 'html.parser')
    # opening a file in write mode
    # note: w creates a file and replaces the content,
    #       to append to a file use r+ if the file exist
    #       or a if you want to create the file and append 
    #       to it.
    f = open("t.txt", "a")
    # traverse paragraphs from soup
    for link in soup.find_all("a"):
        data = link.get('href')
        f.write(data)
        f.write("\n")
    
    f.close()***```

of course you could do more in refactoring this all in functions or use a context manager for the writing in the file but the approach should get you started.

Update:

small change so you use a conttext manager for the file and only greps the links with a .pdf extention.

    import requests
    from bs4 import BeautifulSoup
    
    
    BASE_URL = 'https://www.nzx.com/announcements/'      
    pdf_list = ['190008', '372918', '372980']
    
    with open("t.txt", "a") as f:
        for pdf in pdf_list:
            grab = requests.get(f'{BASE_URL}{pdf}')
            soup = BeautifulSoup(grab.text, 'html.parser')
            for link in soup.find_all("a"):
                data = link.get('href')
                if data.endswith('.pdf'):
                    f.write(data)
                    f.write("\n")

Update 2:

you can also extrat the pdf right away , i leave the t.txt file in this example but it isnt needed if you simply want to download the pdfs

import requests
from bs4 import BeautifulSoup


BASE_URL = 'https://www.nzx.com/announcements/'
pdf_list = ['190008', '372918', '372980']

with open("t.txt", "a") as f:
    for pdf in pdf_list:
        grab = requests.get(f'{BASE_URL}{pdf}')
        soup = BeautifulSoup(grab.text, 'html.parser')
        for link in soup.find_all("a"):
            data = link.get('href')
            if data.endswith('.pdf'):
                fname = data.split('/')[-1]
                pdf = requests.get(data)
                with open(fname, "wb") as pf:
                    pf.write(pdf.content)
                f.write(data)
                f.write("\n")

You need a list. Put all you links into that list. Than you need a for loop to go through all the links. Inside the loop you write code which will work with one link. After it has finished it will take the next and do the same with it until there is no link left in the list.

Related