When I try to scrape content with splash I get an empty list, why?

Viewed 46

Here is the website I'm trying to scrape https://people.sap.com/tim.sheppard

In particular I'm trying to scrape the first post, which on the inspect window has this position:

<div class="dm-content-item__text">We have migrated an application from PB 7 to PB 12.5.2 build 5006. After the migration we're having problems with some computed fields in datawindows. The app has many datawindows that have computed fields which include a date() function using syntax...</div>

my spider is the following

import scrapy
from scrapy_splash import SplashRequest

class RedditSpider(scrapy.Spider):
    name = 'quotes'
    allowed_domains = ['people.sap.com']
    start_urls = ['http://people.sap.com/tim.sheppard']

    def start_requests(self):
        for url in self.start_urls:
            yield SplashRequest(url=url,
                                 callback=self.parse,
                                 endpoint='render.html')

    def parse(self, response):
        quote = response.xpath('//*[@class="dm-content-item__text"]/text()').extract()
        yield {"quote": quote}

and this is the response I get

2021-06-10 17:33:22 [scrapy.core.engine] DEBUG: Crawled (200) <GET http://people.sap.com/tim.sheppard via http://localhost:8050/render.html> (referer: None)
2021-06-10 17:33:22 [scrapy.core.scraper] DEBUG: Scraped from <200 http://people.sap.com/tim.sheppard>
{'quote': []}
2021-06-10 17:33:22 [scrapy.core.engine] INFO: Closing spider (finished)
2021-06-10 17:33:22 [scrapy.statscollectors] INFO: Dumping Scrapy stats:

I don't understand what I'm doing wrong...

0 Answers
Related