Finding the data corresponding to a specific selection in an html form

user_14551574 0
Viewed 50

I am trying to scrape data from the form located at http://appl101.lsu.edu/booklet2.nsf/Selector2?OpenForm The action of the form is "/booklet2.nsf/f5e6e50d1d1d05c4862584410071cd2e?CreateDocument". For each pair (Semester, Department) selected, we get a corresponding page that contains the data.

My goal is to write some python code to find the URL of the page for each pair (Semester, Department). To start with, I was trying to find the URL of a specific selection, say, (Fall 2020, Mathematics).

I am new to web scraping and know only some elementary html. If someone can direct me in the right direction, it would be highly appreciated. Also, please elaborate a bit about the action of this form.

web-scraping forms html python
2 Answers

the action of the form: is just another page on the same server, it just has a strange name

for the scraping part: there is no unique URL for each page corresponding for a pair of (semester, Department). what happens is the semester and department you choose will be submitted to the server in a post request and based on the data submitted the content of the page corresponding the URL in the action of the form will be different (dynamic web page).

solution: one approach is to store the strange url and the pairs of (semester, department) and when you need to get the page corresponding to the pair, send a post request to the URL and provide the the pair (semester, department) within the request in a key value manner and the key would be the name of the "select input and the Department input from the form " then you should receive the page in html which you can scrape it and extract the information you need.

for example [key value for post request] SemesterDesc=Fall 2020 as "SemesterDesc" is the name of the select input for the semester/year in your form, same apply for department input.

you could search for how to make a post request.

user_83631260

You can use this example to get URLs of each pair (but majority will return NoCourseDept):

import requests
from bs4 import BeautifulSoup

base_url = 'http://appl101.lsu.edu/booklet2.nsf/Selector2?OpenForm'
post_url = 'http://appl101.lsu.edu/booklet2.nsf/f5e6e50d1d1d05c4862584410071cd2e?CreateDocument'

soup = BeautifulSoup(requests.get(base_url).content, 'lxml')

semesters = []
for s in soup.select('[name="SemesterDesc"] [value]'):
    semesters.append(s['value'])

departments = []
for d in soup.select('[name="Department"] option'):
    departments.append(d.get_text(strip=True))

for s in semesters:
    for d in departments:
        data = {
            '%%Surrogate_SemesterDesc':1,
            'SemesterDesc':s,
            '%%Surrogate_Department': 1,
            'Department':d
        }
        r = requests.post(post_url, data=data)
        print('{:<30} {:<30} {}'.format(s, d, r.url))

Prints:

...

Second Summer Module 2021      BUSINESS ADMINISTRATION        https://appl101.lsu.edu/booklet2.nsf/All/FFAC316D00E5F3D5862585C7002EF1AA?OpenDocument
Second Summer Module 2021      BUSINESS EDUCATION             https://appl101.lsu.edu/booklet2.nsf/NoCourseDept?readform
Second Summer Module 2021      BUSINESS LAW                   https://appl101.lsu.edu/booklet2.nsf/NoCourseDept?readform
Second Summer Module 2021      CHEMICAL ENGINEERING           https://appl101.lsu.edu/booklet2.nsf/NoCourseDept?readform
Second Summer Module 2021      CHEMISTRY                      https://appl101.lsu.edu/booklet2.nsf/NoCourseDept?readform
Second Summer Module 2021      CHILD AND FAMILY STUDIES       https://appl101.lsu.edu/booklet2.nsf/NoCourseDept?readform
Second Summer Module 2021      CHINESE                        https://appl101.lsu.edu/booklet2.nsf/NoCourseDept?readform
Second Summer Module 2021      CIVIL ENGINEERING              https://appl101.lsu.edu/booklet2.nsf/NoCourseDept?readform
Second Summer Module 2021      CIVIL & ENVIRONMENTAL ENGINEER https://appl101.lsu.edu/booklet2.nsf/NoCourseDept?readform
Second Summer Module 2021      CLASSICAL STUDIES              https://appl101.lsu.edu/booklet2.nsf/NoCourseDept?readform
Second Summer Module 2021      COMMUNICATION DISORDERS        https://appl101.lsu.edu/booklet2.nsf/NoCourseDept?readform
Second Summer Module 2021      COMMUNICATION STUDIES          https://appl101.lsu.edu/booklet2.nsf/NoCourseDept?readform
Second Summer Module 2021      COMPARATIVE BIOMEDICAL SCIENCE https://appl101.lsu.edu/booklet2.nsf/NoCourseDept?readform
Second Summer Module 2021      COMPARATIVE LITERATURE         https://appl101.lsu.edu/booklet2.nsf/NoCourseDept?readform
Second Summer Module 2021      COMPUTER SCIENCE               https://appl101.lsu.edu/booklet2.nsf/NoCourseDept?readform
Second Summer Module 2021      CONSTRUCTION MANAGEMENT        https://appl101.lsu.edu/booklet2.nsf/All/637EAD668A213EDC862585F200296FAE?OpenDocument
Second Summer Module 2021      CURRICULUM & INSTRUCTION       https://appl101.lsu.edu/booklet2.nsf/NoCourseDept?readform
Second Summer Module 2021      DAIRY SCIENCE                  https://appl101.lsu.edu/booklet2.nsf/NoCourseDept?readform
Second Summer Module 2021      DIGITAL MEDIA ARTS & ENGINEERI https://appl101.lsu.edu/booklet2.nsf/NoCourseDept?readform
Second Summer Module 2021      DISASTER SCIENCE MANAGEMENT    https://appl101.lsu.edu/booklet2.nsf/NoCourseDept?readform
Second Summer Module 2021      DOCTOR OF DESIGN               https://appl101.lsu.edu/booklet2.nsf/NoCourseDept?readform
Second Summer Module 2021      ECONOMICS                      https://appl101.lsu.edu/booklet2.nsf/NoCourseDept?readform
Second Summer Module 2021      EDUC LEADERSHIP RESEARCH COUNS https://appl101.lsu.edu/booklet2.nsf/All/B0D27015A5F630CF86258602002C263E?OpenDocument
Second Summer Module 2021      EDUCATION                      https://appl101.lsu.edu/booklet2.nsf/NoCourseDept?readform

...
user_100359850
Related