Passing page source obtained from selenium to scrapy

Question

I am trying to implement a scrapy + selenium crawler but I can't seem to get scrapy to parse the HTML after I interact with the page using selenium.

The code is as follows

I guess I do not have to make the requests using scrapy.Request that is my guess but I have not found any other way.

import scrapy
import selenium
from Clasipar.items import AnuncioItem
from selenium import webdriver
import time
from selenium.webdriver.common.by import By
from selenium.webdriver.support.ui import WebDriverWait as wait
from selenium.webdriver.support import expected_conditions as EC


class ClasiparSpider(scrapy.Spider):
    name = "test"

    driver = webdriver.Firefox()

    def start_requests(self):
        urls = ['https://clasipar.paraguay.com/inmuebles/casas/hermosa-casa-enbarrio-los-mangales-1142381']

        for url in urls:
            self.driver.get(url)
            self.driver.find_element_by_id('btn_view_info').click()
            self.driver.implicitly_wait(2)
            wait(self.driver, 10).until_not(EC.text_to_be_present_in_element((By.XPATH, '//*[@id="phone_number"]'), "*********"))
            yield scrapy.Request(url=self.driver.page_source,callback=self.parse)

    def parse(self, response):
        item = AnuncioItem()
        titulo = response.XPATH(
            '/html/body/main/div/section/span').extract().strip()
        anunciante = response.XPATH(
            '/html/body/main/div/section/div[3]/div[1]/div/h5').extract().strip()
        telefono = response.XPATH('//*[@id="phone_number"]').extract().strip()
        email = response.XPATH(
            '/html/body/main/div/section/div[3]/div[2]/form/ul/li[2]/b/a').extract().strip()
        precio = response.XPATH(
            '/html/body/main/div/section/div[3]/div[3]/h3').extract().strip()
        departamento = response.XPATH(
            '/html/body/main/div/section/div[4]/div[1]/h6').extract().strip()
        nroAnuncio = response.XPATH(
            '/html/body/main/div/section/div[4]/div[2]/h6').extract().strip()
        zona = response.XPATH(
            '/html/body/main/div/section/div[4]/div[3]/h6').extract().strip()
        visitas = response.XPATH(
            '/html/body/main/div/section/div[4]/div[4]/h6').extract().strip()
        fechaPublicacion = response.XPATH(
            '/html/body/main/div/section/div[4]/div[5]/h6').extract().strip()
        descripcion = response.XPATH(
            '/html/body/main/div/section/p').extract().strip()
    pass

user_6315345 · Answer

I think there is no way, but I'm not really sure.

I searched for quite a while and found nothing.

I'm having to do the same thing at work, here we use scrapy but the website I'm trying to scrape uses ajax and javascript to load new data and there is no way for me to get the data by any other means other then 'clicking' a button.

So the 'best fix' I found is using beautiful soup to parse the HTML, and then I call a scrapy request to each item I'm scraping.

It's not ideal, but it does the job. Even though it is slower than just using beautiful soup, it takes advantages of all the neat things scrapy provides.

Passing page source obtained from selenium to scrapy

1 Answers