Scrapy: How to send null value in FormRequest.from_response{}'s formdata attibute?

user_10923739 0
Viewed 78

I am trying to send form data using FormRequest.From_reponse() method. But the problem is there are some dummy values and some blank value in the form data. So, for null values I am sending '' or None and 'space' but nothing seem to work,

 form_dta = {
                    'term_in': '202130',
                    'sel_subj': 'dummy',
                    'sel_subj': 'CS',
                    'SEL_CRSE': '100',
                    'SEL_TITLE': '',
                    'BEGIN_HH': '0',
                    'BEGIN_MI': '0',
                    'BEGIN_AP': 'a',
                    'SEL_DAY': 'dummy',
                    'SEL_PTRM': 'dummy',
                    'END_HH': '0',
                    'END_MI': '0',
                    'END_AP': 'a',
                    'SEL_CAMP': 'dummy',
                    'SEL_SCHD': 'dummy',
                    'SEL_SESS': 'dummy',
                    'SEL_INSTR': 'dummy',
                    'SEL_INSTR': '%',
                    'SEL_ATTR': 'dummy',
                    'SEL_ATTR': '%',
                    'SEL_LEVL': 'dummy',
                    'SEL_LEVL': '%',
                    'SEL_INSM': 'dummy',
                    'sel_dunt_code': '',
                    'sel_dunt_unit': '',
                    'call_value_in': '',
                    'rsts': 'dummy',
                    'crn': 'dummy',
                    'path': '1',
                    'SUB_BTN': 'View+Sections'
        }
        yield scrapy.FormRequest.from_response(
            response,
            formcss='div.pagebodydiv form',
            formdata=form_dta,
            callback=self.course_scrap)

Sending '' for as suggesting null value to form request does not work. Form actually only contains 1 select tag on frontend where I have to select course name(name=sel_subj). all other form data is hidden.

scrapy web-scraping request python
1 Answers

I meet the same problem.

At first, I tried '' like you, and failed. Then I use None, but python tell me that to_bytes must receive a str or bytes.

After checking the source code, I found that by default FormRequest will encode the formdata with to_bytes() function and then set the headers "Content-Type" to "application/x-www-form-urlencoded".

What you are looking for is JsonRequest which use json.dumps as its encode function. None in python will convert into null in javascript as you wish.

from scrapy.http.request.json_request import JsonRequest

Be careful, JsonRequest needs parameter data instead of formdata.

English is not my native language, sorry if there is any mistakes.

user_196122770
Related