Getting incorrect link on parsing web page in BeautifulSoup

Question

I'm trying to get the download link from the button in this page. But when I open the download link that I get from my code I get this message

I noticed that if I manually click the button and open the link in a new page the csrfKey part of the link is always same whereas when I run the code I get a different key every time. Here's my code

from bs4 import BeautifulSoup
import requests
import re

def GetPage(link):
    source_new = requests.get(link).text
    soup_new = BeautifulSoup(source_new, 'lxml')
    container_new = soup_new.find_all(class_='ipsButton')
    for data_new in container_new:
        #print(data_new)
        headline = data_new # Display text
        match = re.findall('download', str(data_new), re.IGNORECASE)
        if(match):
            print(f'{headline["href"]}
')


if __name__ == '__main__':
    
    link = 'https://eci.gov.in/files/file/10985-5-number-and-types-of-constituencies/'
    GetPage(link)

user_6106791 · Answer

Before you get to the actual download links of the files, you need to agree to Terms and Conditions. So, you need to fake this with requests and then parse the next page you get.

Here's how:

import requests
from bs4 import BeautifulSoup

if __name__ == '__main__':
    link = 'https://eci.gov.in/files/file/10985-5-number-and-types-of-constituencies/'
    with requests.Session() as connection:
        r = connection.get("https://eci.gov.in/")

        confirmation_url = BeautifulSoup(
            connection.get(link).text, 'lxml'
        ).select_one(".ipsApp .ipsButton_fullWidth")["href"]

        fake_agree_to_continue = connection.get(
            confirmation_url.replace("?do=download", "?do=download&confirm=1")
        ).text

        download_links = [
            a["href"] for a in
            BeautifulSoup(
                fake_agree_to_continue, "lxml"
            ).select(".ipsApp .ipsButton_small")[1:]]

        for download_link in download_links:
            response = connection.get(download_link)
            file_name = (
                response
                .headers["Content-Disposition"]
                .replace('"', "")
                .split(" - ")[-1]
            )
            print(f"Downloading: {file_name}")
            with open(file_name, "wb") as f:
                f.write(response.content)

This should output:

Downloading: Number And Types Of Constituencies.pdf
Downloading: Number And Types Of Constituencies.xls

And save two files: a .pdf and a .xls. The later one looks like this:

Getting incorrect link on parsing web page in BeautifulSoup

1 Answers