Scrapy Spider Doesn't Return Any Information

Viewed 83

I'm a student, and for a project I'm collecting information on brands. I found this website called Kit: Kit Page that I want to scrape for brands. It has almost 500 pages, and I wrote a Scrapy Spider in Python 3 that go through each of the pages and copies the list to a dictionary, but I can't figure out the xpath or css to actually get the list info. Here's my items.py:

import scrapy

class KitcreatorwebscraperItem(scrapy.Item):
    creator = scrapy.Field()

and here's my spider:

import scrapy

class KitCreatorSpider(scrapy.Spider):
    name = "kitCreators"
    pageNumber = 1

    start_urls = [
        'https://kit.com/brands?page=1',
    ]

    while pageNumber <= 478:
        newUrl = "https://kit.com/brands?page=" + str(pageNumber)
        start_urls.append(newUrl)
        pageNumber += 1

    def parse(self, response):
        for li in response.xpath('//div[@class="section group"][0]'):

It runs successfully, but I have been unable to write an xpath that gets the data I need. What path is necessary, and how do I implement that in the code?

1 Answers
Related