Scrape next page which is an php action

Viewed 47

I have been working on a webscraper for a website which shows accidents. With help I accomplished to scrape the first page, but I am having trouble to scrape the next pages. They are accidents which happend earlier. I used Scrapy and Selenium for this because the website needs to load for a few seconds.

This is what I have so far:

import scrapy
from scrapy import Selector
from scrapy_selenium import SeleniumRequest
#from selenium.common.exceptions import NoSuchElementException

class MeldingenSpider(scrapy.Spider):
    name = 'meldingen'

    responses = []

    def start_requests(self):
        yield SeleniumRequest(
            url='http://ftp.112meldingen.nl/index.php',
            callback=self.parse
        )

    def parse(self, response):
        driver = response.meta['driver']
        intial_page = driver.page_source
        self.responses.append(intial_page)
        driver.implicitly_wait(3)

        for resp in self.responses:
            r = Selector(text=resp)
            articles = r.css('table#alerts')
            for article in articles:
            #if "haven" in article.css('div.title a::text').get():
                    yield {
                        'headline': article.css('td.bold a::text').get() ,
                        'timestamp': article.css('td.bold span::text').get().replace('\xa0\xa021',''),
                        'eenheden' : [x.replace('\xa0',' ') for x in article.xpath('.//tr/td[@class="bold center"]/following-sibling::td//text()').getall()][-1],
                        'urgentie' : article.css('td > td').get()[3]
                    }
        next_page = response.css('form::attr(action)').get()
        if next_page is not None:
            yield response.follow(next_page, callback=self.parse)

I used the code in the bottem for going to the next page in scrapers which did not need Selenium. Since the accidents go way back, I only need them 2021 so there needs to be an if-statement which can control the timestamp. I am quite new to this and really eager to learn how this works, so any help we be appriciated!

0 Answers
Related