How to use Python requests to fill out a date option as well as download from a radiobutton

user_11567495 0
Viewed 148

I'm trying to make a python script that will scrape data from this website

https://noms.wei-pipeline.com/reports/ci_report/launch.php?menuitem=2600315

and download the CSV from yesterday. As you can see, it's got two menu options for dates, a radio button for CSV and then a submit button.

I thought perhaps I could use the requests library? Not looking for someone to do it for me, but if anyone could point me in the right direction that would be great!

I know this is too simple but here is what I have so far:

import requests

print('Download Starting...')

url = 'https://noms.wei-pipeline.com/reports/ci_report/launch.php?menuitem=2600315'

r = requests.get(url)

filename = url.split('/')[-1] # this will take only -1 splitted part of the url

with open(filename,'wb') as output_file:
    output_file.write(r.content)

print('done')
python-requests web-scraping python
1 Answers

You need first to use requests.Session() in order to store cookies and re-send them in subsequent requests. The process is the following :

  • get the original URL first to get the cookies (session id)

  • make a request on POST /reports/ci_report/server/request.php with some parameters including date and output format. The result is a json with an id like this :

    {'jrId': 'jr_13879611'}

  • make a request on GET /reports/ci_report/server/streamReport.php?jrId=jr_13879611 which gives the csv data

There is a parameter in the POST request where we need the menuitem query param value from your original url, so we parse the query params to get it using urlparse :

import requests
import time
import urllib.parse as urlparse
from urllib.parse import parse_qs
from datetime import datetime,timedelta

yesterday = datetime.now() - timedelta(1)
yesterday_date = f'{yesterday.strftime("%d")}-{yesterday.strftime("%B")[:3]}-{yesterday.strftime("%Y")}'

original_url = "https://noms.wei-pipeline.com/reports/ci_report/launch.php?menuitem=2600315"
parsed = urlparse.urlparse(original_url)

target_url = "https://noms.wei-pipeline.com/reports/ci_report/server/request.php"
stream_report_url = "https://noms.wei-pipeline.com/reports/ci_report/server/streamReport.php"

s = requests.Session()
# load the cookies
s.get(original_url)

#get id
r = s.post(target_url,
    params = {
        "request.preventCache": int(round(time.time() * 1000))
    },
    data = {
        "ReportProc": "CIPR_DAILY_BULLETIN",
        "p_ci_id": parse_qs(parsed.query)['menuitem'][0],
        "p_opun": "PL",
        "p_gas_day_from": yesterday_date,
        "p_gas_day_to": yesterday_date,
        "p_output_option": "CSV"
})
r = s.get(stream_report_url, params = r.json())
print(r.text)

Try this on repl.it

user_26143640
Related