Download file by redirecting to general URL from profile URL

user_17081630 0
Viewed 31

I have a situation where the download link for a website is generalized across all individual profiles: https://www.ffiec.gov/npw/Institution/HistoryExportToCSV

Which appears to be implemented by visiting after visiting the profile URL: https://www.ffiec.gov/npw/Institution/Profile/242?dt=20120101

I'm trying to handle this using the requests library, but is there a way to sequentially visit the first URL after visiting the second?

I thought the following might do it but I'm not getting anything in return:

import requests

headers = {"Referer": "https://www.ffiec.gov/npw/Institution/Profile/242?dt=20120101"}
url = "https://www.ffiec.gov/npw/Institution/HistoryExportToCSV"

req = requests.get(url, headers=headers)
csv = req.content
python-requests python
1 Answers

It's easier to load the data in Json format (instead of CSV):

import requests

history_url = "https://www.ffiec.gov/npw/Institution/LoadHistory"
payload = {"RssdID": "242"} # the number is from profile URL

data = requests.post(history_url, data=payload)
print(data.json())

Prints:

[
    {
        "Event": "Address Changed",
        "EventDate": "3/31/2006",
        "OtherInstitutionName": None,
        "OtherInstitutionID": None,
        "Notes": "The address of FIRST COMMUNITY BANK XENIA-FLORA changed from FRONT STREET  to 260 FRONT STREET .",
    },
    {
        "Event": "Name Changed",
        "EventDate": "10/30/2001",
        "OtherInstitutionName": None,
        "OtherInstitutionID": None,
        "Notes": "FIRST NATIONAL BANK OF XENIA, THE changed to the new name FIRST COMMUNITY BANK XENIA-FLORA.",
    },
    {
        "Event": "Institution Type Changed",
        "EventDate": "10/30/2001",
        "OtherInstitutionName": None,
        "OtherInstitutionID": None,
        "Notes": "FIRST COMMUNITY BANK XENIA-FLORA changed from National Bank to State Member Bank.",
    },
    {
        "Event": "Institution Established",
        "EventDate": "1/1/1922",
        "OtherInstitutionName": None,
        "OtherInstitutionID": None,
        "Notes": "FIRST NATIONAL BANK OF XENIA, THE was established as a National Bank at FRONT STREET , XENIA, IL.",
    },
]
user_100359850
Related