Unable to click on the next button in pagination

user_14099429 0
Viewed 284

I'm using scrapy with scrapy-selenium and I'm unable to handle pagination because href contains only # symbol.

class PropertyScraperSpider(scrapy.Spider):
    name = 'property_scraper'
    allowed_domains = ['www.samtrygg.se']



    def start_requests(self):
        yield SeleniumRequest(
            url='https://www.samtrygg.se/RentalObject/NewSearch',
            wait_time=3,
            headers=self.headers,
            callback=self.parse_links
        )
        

    def parse_links(self, response):
        cards = response.xpath("//div[@class='owl-carousel owl-theme show-nav-hover']/div/a")

        for card in cards:
            link = card.xpath(".//@href").get()

            print('\n\n:link',len(link))

            yield SeleniumRequest(
                url= link,
                wait_time=3,
                headers=self.headers,
                callback=self.parse,
            )
        next_page = response.xpath("//a[@id='next']/@href").get()

        print('\n\n\nNEXT_PAGE',next_page)
        if next_page:
            absolute_url = f'https://www.samtrygg.se/RentalObject/NewSearch{next_page}'
            yield SeleniumRequest(
                url=absolute_url,
                headers=self.headers,
                wait_time=3,
                callback=self.parse_links
            )
            
    def parse(self,response):
        pass

I need help with this pagination issue. how can I handle it? Any help would be highly appreciated.

scrapy web-scraping selenium python
4 Answers

Approaching Dynamic Content in Scrapy

What Ryan is saying is correct. To expand on this, dynamic content can be grabbed in a few ways.

  1. By re-engineering HTTP requests

This is by far the best way to grab dynamic content if possible, its the most efficient and less brittle than selenium. This is based on whether the javascript is triggering an HTTP request to grab data for the webpage. In this case it is and should be tried first before resorting to other means

  1. Using Splash (Browser Activity)

Scrapy has a middleware which integrates splash. Splash pre-renders pages so enables access to javascript loaded HTML. It also has some browser activity functionality. Less labour intensive than selenium but still it's browser activity.

  1. Using selenium_scrapy (Browser Activity)

This is the solution you are trying here, the problem is, it doesn't really give a lot of options to do complex browser activity. So it's real purpose is in being able to grab HTML that has been javascript loaded really.

  1. Using selenium in the middlewares (Browser Activity)

You can use middlewares to filter the requests, using the full selenium package. This is okay for when there is no alternative and you want something for every request. Or you want to customise it based on the type of request you're making

  1. Using selenium straight in the spider script. (Browser Activity)

This is the last resort in my opinion when all other options are not available and you really need complex browser activity for specific parts of your script and can't blanket do it using requests.

Re-engineering the requests

So now you have a basic understanding of what it is. Your browser (I prefer chrome) has access to all the requests the browser makes to display the site you see. If you inspect the page --> network tools --> XHR you will see all the AJAX requests (typically where the API endpoints live).

enter image description here

You can see all the request, sorting by size tends to work as typically the data will be a larger request. When you click the request, you get access to the headers it sends, a preview of the response and the response.

enter image description here

So here we have the preview of the data you probably want. I will then copy this request in the form of cURL and input it into a website like curl.trillworks.com.

enter image description here

This gives you the headers, parameters and cookies if necessary to make the correct Scrapy Request. In this case, you actually only need one of the parameters to mimic the HTTP request. I tend to use the requests package to play about with what I actually need as copying the request gives you everything in the request, some if which you wont need.

user_125660460

The website is using an API that is visible if you look at the requests made by your web browser when you open https://www.samtrygg.se/RentalObject/NewSearch

API URL: https://www.samtrygg.se/RentalObject/SearchResult?search=sverige&neLat=&neLng=&swLat=&swLng=

You could just make a single request to the API URL with Scrapy to get all the listings.

It seems that the website doesn't have any actual "Pagination". It just loads all the data on the first request and then does some frontend manipulation to show a partial amount of the results depending on the "page" that the user is on.

user_48764930

I checked if there's an API and I didn't find it.

So in that case, if you're using Selenium you need to check if the next page button is available, if yes then you click on it and then insert the HTML markup to an array.

Example:


responses = []
next = driver.find_elements_by_xpath("XPATH")
while len(next) > 0:
    next.click()
    responses.append(driver.page_source)

Kind regards, Ahmed

user_107059080 
import scrapy
import json
 
NIFTY_FIFTY = "https://www.samtrygg.se/RentalObject/SearchResult?search=sverige&neLat=&neLng=&swLat=&swLng="
 
 
class LiveSpider(scrapy.Spider):
    name = "esos_nortes"
    start_urls = [NIFTY_FIFTY]
    allowed_domains = ["www.samtrygg.se"]
 
    # Custom Settings are needed to send the User Agent.         
    custom_settings = {
        'USER_AGENT' : 'Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/81.0.4044.138 Safari/537.36'
    }
 
    def parse(self, response):
        json_response = json.loads(response.body.decode("utf-8"))
        
        # We want the full first 25 addresses, for example:
        for firsts_25 in range(24):
            print(json_response['SearchResult'][firsts_25]['FullAddress'])

NIFTY_FIFTY url is obtained as explained by AaronS, observing the tools of your browser

user_62531000
Related