Downloading Pdf into Subdirectory

user_7361574 0
Viewed 82

I'm trying to get the PDFs to download into a subdirectory using the form name so it should look like Form W-2/Form W-2 2020. Currently, it just downloads to the same folder as the main app.

        pdf_link = form_number.find("a")
        i += 1
        print("Downloading file: ", i)
        response = requests.get(pdf_link.get('href'))
        pdf = open(form_number.text.strip() + "-" + form_year.text.strip() + ".pdf", 'wb')
        pdf.write(response.content)
        pdf.close()
        print("File ", i, " downloaded")
web-scraping beautifulsoup pdf python
1 Answers

You can use os.path.join to join path components, os.path.exists to check if directory exists and os.makedirs to create the directory.

This example combines the methods:

import os
import requests
from bs4 import BeautifulSoup

form = "Form W-2"
URL = (
    "https://apps.irs.gov/app/picklist/list/priorFormPublication."
    "html?resultsPerPage=200&sortColumn=sortOrder&indexOfFirstRow=0&criteria=formNumber&value="
    "" + form + "&isDescending=false"
)
page = requests.get(URL)

soup = BeautifulSoup(page.content, "html.parser")
for table_element in soup.select(".picklist-dataTable tr:has(td)"):
    form_number = table_element.find("td", class_="LeftCellSpacer")
    u = form_number.a["href"]
    path = os.path.join(form, u.split("/")[-1])

    if not os.path.exists(form):
        os.makedirs(form)

    print(f"Saving {u=} to {path=}")
    with open(path, "wb") as f_out:
        f_out.write(requests.get(u).content)

Prints:

Saving u='https://www.irs.gov/pub/irs-prior/fw2p--1990.pdf' to path='Form W-2/fw2p--1990.pdf'
Saving u='https://www.irs.gov/pub/irs-prior/fw2p--1989.pdf' to path='Form W-2/fw2p--1989.pdf'
Saving u='https://www.irs.gov/pub/irs-prior/fw2p--1988.pdf' to path='Form W-2/fw2p--1988.pdf'

...and so on.

and saves the documents to directory.

EDIT: To save with different filenames:

import os
import requests
from bs4 import BeautifulSoup

form = "Form W-2"
URL = (
    "https://apps.irs.gov/app/picklist/list/priorFormPublication."
    "html?resultsPerPage=200&sortColumn=sortOrder&indexOfFirstRow=0&criteria=formNumber&value="
    "" + form + "&isDescending=false"
)
page = requests.get(URL)

soup = BeautifulSoup(page.content, "html.parser")
for table_element in soup.select(".picklist-dataTable tr:has(td)"):
    form_number = table_element.find("td", class_="LeftCellSpacer")
    form_year = table_element.find("td", class_="EndCellSpacer")
    u = form_number.a["href"]
    p = "{}-{}.pdf".format(
        form_number.get_text(strip=True), form_year.get_text(strip=True)
    )

    path = os.path.join(form, p)

    if not os.path.exists(form):
        os.makedirs(form)

    print(f"Saving {u=} to {path=}")
    with open(path, "wb") as f_out:
        f_out.write(requests.get(u).content)

This saves the files as:

Saving u='https://www.irs.gov/pub/irs-prior/fw2p--1990.pdf' to path='Form W-2/Form W-2 P-1990.pdf'
Saving u='https://www.irs.gov/pub/irs-prior/fw2p--1989.pdf' to path='Form W-2/Form W-2 P-1989.pdf'
Saving u='https://www.irs.gov/pub/irs-prior/fw2p--1988.pdf' to path='Form W-2/Form W-2 P-1988.pdf'
Saving u='https://www.irs.gov/pub/irs-prior/fw2p--1987.pdf' to path='Form W-2/Form W-2 P-1987.pdf'
Saving u='https://www.irs.gov/pub/irs-prior/fw2p--1986.pdf' to path='Form W-2/Form W-2 P-1986.pdf'

...
user_100359850
Related