I have been working on a webscraper for a website which shows accidents. With help I accomplished to scrape the first page, but I am having trouble to scrape the next pages. They are accidents which happend earlier. I used Scrapy and Selenium for this because the website needs to load for a few seconds.
This is what I have so far:
import scrapy
from scrapy import Selector
from scrapy_selenium import SeleniumRequest
#from selenium.common.exceptions import NoSuchElementException
class MeldingenSpider(scrapy.Spider):
name = 'meldingen'
responses = []
def start_requests(self):
yield SeleniumRequest(
url='http://ftp.112meldingen.nl/index.php',
callback=self.parse
)
def parse(self, response):
driver = response.meta['driver']
intial_page = driver.page_source
self.responses.append(intial_page)
driver.implicitly_wait(3)
for resp in self.responses:
r = Selector(text=resp)
articles = r.css('table#alerts')
for article in articles:
#if "haven" in article.css('div.title a::text').get():
yield {
'headline': article.css('td.bold a::text').get() ,
'timestamp': article.css('td.bold span::text').get().replace('\xa0\xa021',''),
'eenheden' : [x.replace('\xa0',' ') for x in article.xpath('.//tr/td[@class="bold center"]/following-sibling::td//text()').getall()][-1],
'urgentie' : article.css('td > td').get()[3]
}
next_page = response.css('form::attr(action)').get()
if next_page is not None:
yield response.follow(next_page, callback=self.parse)
I used the code in the bottem for going to the next page in scrapers which did not need Selenium. Since the accidents go way back, I only need them 2021 so there needs to be an if-statement which can control the timestamp. I am quite new to this and really eager to learn how this works, so any help we be appriciated!