Here is the website I'm trying to scrape https://people.sap.com/tim.sheppard
In particular I'm trying to scrape the first post, which on the inspect window has this position:
<div class="dm-content-item__text">We have migrated an application from PB 7 to PB 12.5.2 build 5006. After the migration we're having problems with some computed fields in datawindows. The app has many datawindows that have computed fields which include a date() function using syntax...</div>
my spider is the following
import scrapy
from scrapy_splash import SplashRequest
class RedditSpider(scrapy.Spider):
name = 'quotes'
allowed_domains = ['people.sap.com']
start_urls = ['http://people.sap.com/tim.sheppard']
def start_requests(self):
for url in self.start_urls:
yield SplashRequest(url=url,
callback=self.parse,
endpoint='render.html')
def parse(self, response):
quote = response.xpath('//*[@class="dm-content-item__text"]/text()').extract()
yield {"quote": quote}
and this is the response I get
2021-06-10 17:33:22 [scrapy.core.engine] DEBUG: Crawled (200) <GET http://people.sap.com/tim.sheppard via http://localhost:8050/render.html> (referer: None)
2021-06-10 17:33:22 [scrapy.core.scraper] DEBUG: Scraped from <200 http://people.sap.com/tim.sheppard>
{'quote': []}
2021-06-10 17:33:22 [scrapy.core.engine] INFO: Closing spider (finished)
2021-06-10 17:33:22 [scrapy.statscollectors] INFO: Dumping Scrapy stats:
I don't understand what I'm doing wrong...