Scrapy form not submitting properly

Question

I want to submit the form with the 5 data that's on the below. By submitting the form, I can get the redirection URL. I don't know where is the issue. Can anyone help me to submit the form with required info. to get the next page URL.

Code for your reference:

import requests
import scrapy


class QuotesSpider(scrapy.Spider):
    name = "uksite"
    login_url = 'https://myeplanning.oxfordshire.gov.uk/Disclaimer/Accept?returnUrl=%2FSearch%2FAdvanced'
    start_urls = [login_url]
    def parse(self, response):
        token = response.css('input[name="__RequestVerificationToken"]::attr(value)').extract_first()
        data = {'__RequestVerificationToken': token,
                'DateReceivedFrom': '2021-04-07',
                'DateReceivedTo': '2021-04-08',
                'AdvancedSearch': 'True',
                'SearchPlanning': 'True',
                }


        yield scrapy.FormRequest.from_response(response,
                                               url=self.login_url,
                                               formdata= data,
                                               clickdata={'class': 'occlss-button occlss-button--primary decompress'},
                                               callback = self.value,
                                               )
    def value(self, response):
   
        print(response._url)

INPUT URL ==> https://myeplanning.oxfordshire.gov.uk/Disclaimer/Accept?returnUrl=%2FSearch%2FAdvanced

Output URL for the given input ==> https://myeplanning.oxfordshire.gov.uk/Planning/Display/MW.0047/21

user_9189799 · Answer

Okay, this should do it.

class MyePlanningSpider(scrapy.Spider):
    name = "myeplanning"
    start_urls = ['https://myeplanning.oxfordshire.gov.uk/Disclaimer/Accept?returnUrl=%2FSearch%2FAdvanced']
    login_url = 'https://myeplanning.oxfordshire.gov.uk/Search/Results'

    def parse(self, response):
        data = {
            '__RequestVerificationToken': response.css('input[name="__RequestVerificationToken"]::attr(value)').get(),
            'DateReceivedFrom': '2021-04-07',
            'DateReceivedTo': '2021-04-08',
            'AdvancedSearch': 'True',
            'SearchPlanning': 'True',
        }

        yield scrapy.FormRequest(
            url=self.login_url,
            formdata= data,
            callback=self.parse_value,
        )

    def parse_value(self, response):
        print(response.url)

Scrapy form not submitting properly

1 Answers