I'm trying to parse the title of different listings from this webpage. The titles are not dynamic as they are available in page source. However, it is necessary to send cookies in the first place to grab the titles. I've tried with the following way to scrape the titles of the listings but it doesn't seem to work.
My attempt so far:
import scrapy
from scrapy.crawler import CrawlerProcess
class ControllerSpider(scrapy.Spider):
name = 'controller'
start_urls = [
'https://www.controller.com/listings/aircraft/for-sale/list?SortOrder=23&scf=False&page=1'
]
def start_requests(self):
for i,url in enumerate(self.start_urls):
yield scrapy.Request(url,meta={'cookiejar': i},callback=self.parse)
def parse(self,response):
for item in response.css(".listing-name > a[href]::text").getall():
yield {"title":item}
if __name__ == "__main__":
c = CrawlerProcess({
'USER_AGENT': 'Mozilla/5.0 (Windows NT 6.1; ) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/83.0.4103.116 Safari/537.36',
})
c.crawl(ControllerSpider)
c.start()
How can I grab the titles of different listings from that webpage making use of cookies?
PS I do not wish to hardcode the cookies.