For example, when i use the site 'https://en.zalando.de/men-shoes-boots/', only 37 items are scraped. I understand there needs to be some type of pagination to get to the next page, but on the first page itself there are far more than 37 items, and i cannot seem to figure out why the script stops at 37.
any help would be super appreciated!!
My the finder of elements of my spider looks like this:
class ZalandoSpider (scrapy.Spider):
name = 'zalando'
start_urls = [
'https://en.zalando.de/men-shoes-boots/'
]
def parse(self,response):
alle_elementen = response.css('.DvypSJ')
print('length=' + str(len(alle_elementen)))
Might have something to do with infinite scroll but I have been checking out the Network page for over 2 hours now without result.
Thanks in advance