I have a json file that I want to scrape: https://www.website.com/api/list?limit=50&page=1
I can use 'scrapy.Spider' to crawl all the pages no problem, but if it's possible I prefer to do it with 'CrawlSpider'.
I tried to use:
start_urls=['https://www.website.com']
rules = (
Rule(LinkExtractor(allow=r'/api/list\?.+page=\d+'), callback='parse_page', follow=True),
)
and (just to see if it's even getting the first page):
start_urls=['https://www.website.com']
rules = (
Rule(LinkExtractor(allow=r'/api/list'), callback='parse_page', follow=True),
)
and none of them worked.
Is there a way to do it with 'CrawlSpider'?