scrapy : How to scrape

Question

I'm learning how to scrape using scrapy api.

I would like to scrape the text into the

and the the link into but it's not working (the attached file)

html page

import scrapy

class PriceSpider(scrapy.Spider):
    name = "annonce"  #name of spider

    def start_requests(self):
        urls = [
            'https://www.leboncoin.fr/ventes_immobilieres/offres/ile_de_france/?th=1',

        ]
        for url in urls:
            yield scrapy.Request(url=url, callback=self.parse)

    def parse(self, response):
        for annonce in response.css('section.tabsContent li').extract():
            yield{
                'title':annonce.css('a ::title').extract_first(),
                }

user_9189799 · Answer

Give this a try. Your css selector is heavily flawed.

import scrapy

class PriceSpider(scrapy.Spider):
    name = "annonce"  #name of spider

    def start_requests(self):
        urls = [
            'https://www.leboncoin.fr/ventes_immobilieres/offres/ile_de_france/?th=1',

        ]
        for url in urls:
            yield scrapy.Request(url=url, callback=self.parse)

    def parse(self, response):
        for annonce in response.css('.list_item'):
            yield{
                'link':annonce.css('::attr(href)').extract_first(),
                'title':annonce.css('.item_title::text').extract_first().strip(),
                }

One more thing. Open your settings.py file and make it:

ROBOTSTXT_OBEY = False

scrapy : How to scrape <ul> <li>

1 Answers