How to set a default value when Scrapy selector with extract() returns None?

user_7117003 0
Viewed 1224

I am trying to yield the value of a tag that isn't always present in the pages that I scrape with Scrapy. I am using the extract() function rather than extract_first(). Therefore I cannot seem to set a default value, like suggested in this SO post.

This doesn't work:

def parse(self, response):
        yield {
          'comments': response.css('[itemprop=commentCount]::attr(content)').extract(default=None)
          }

How can I set None as default when I want to use extract() rather than extract_first()?

Thanks very much in advance!

scrapy web-scraping css-selectors python
2 Answers

Try this syntax:

{'comments': response.css('[itemprop=commentCount]::attr(content)').extract() or None}

If result of response.css(CSS) is empty list, then None will be assigned as value of comments key. Otherwise, actual value will be assigned

user_45495540

.extract() yields the output as a list and .extract_first() yields a string.

response.xpath('xpath_of_the_component').extract_first(default="default_value").split()

This line of code will again convert the string to a list and set the default value, if not available.

user_124277820
Related