Why does my spider extract a limited amount of items

Viewed 49

For example, when i use the site 'https://en.zalando.de/men-shoes-boots/', only 37 items are scraped. I understand there needs to be some type of pagination to get to the next page, but on the first page itself there are far more than 37 items, and i cannot seem to figure out why the script stops at 37.

any help would be super appreciated!!

My the finder of elements of my spider looks like this:

class ZalandoSpider (scrapy.Spider):
    name = 'zalando'
    start_urls = [
        'https://en.zalando.de/men-shoes-boots/'
    ]

def parse(self,response):

    alle_elementen = response.css('.DvypSJ')
    print('length=' + str(len(alle_elementen)))

Might have something to do with infinite scroll but I have been checking out the Network page for over 2 hours now without result.

Thanks in advance

0 Answers
Related