CSS selector or XPath that gets information between two i tags?

user_13882421 0
Viewed 153

I'm trying to scrape price information, and the HTML of the website looks like this

<span class="def-price" datasku='....'>
   <i>$</i>
   "999"
   <i>.00<i>
</span>

I want to get 999. (I don't want the dollar sign or the .00) I currently have

product_price_sn = product.css('.def-price i').extract()

I know it's wrong but not sure how to fix it. Any idea how to scrape that price information? Thanks!

scrapy web-scraping web-crawler xpath css
3 Answers

You can use this xpath //span[@class="def-price"]/text()

Make sure you are using /text() and not //text(). Otherwise it will return all text nodes inside span tag.

or

This css selector .def-price::text. When using css selector don't use .def-price ::text, it will return all text nodes like the //text() in xpath.

Using scrapy response.xpath object

from scrapy.http import Request, HtmlResponse as Response

content = '''<span class="def-price" datasku='....'>
   <i>$</i>
   "999"
   <i>.00<i>
</span>'''.encode('utf-8')

url = 'https://stackoverflow.com/questions/62849500'

''' mocking scrapy request object '''
request = Request(url=url)

''' mocking scrapy response object '''
response = Response(url=url, request=request, body=content)

''' using xpath  '''

print(response.xpath('//span[@class="def-price"]/text()').extract())
# outputs ['\n   ', '\n   "999"\n   ']

print(''.join(response.xpath('//span[@class="def-price"]/text()').extract()).strip())
# outputs "99"

''' using css selector '''

print(response.css('.def-price::text').extract())
# outputs ['\n   ', '\n   "999"\n   ']

print(''.join(response.css('.def-price::text').extract()).strip())
# outputs "99"

See it in action here

Using lxml html parser

from lxml import html

parser = html.fromstring("""
<span class="def-price" datasku='....'>
   <i>$</i>
   "999"
   <i>.00<i>
</span>
"""
)

print(parser.xpath('//span[@class="def-price"]/text()'))
# outputs ['\n   ', '\n   "999"\n   ']

print(''.join(parser.xpath('//span[@class="def-price"]/text()')).strip())
# outputs "999"

See it in action here

user_30913980

With BeautifulSoup, you can use CSS selector .def_price and then .find_all(text=True, recursive=0) to get all immediate text.

For example:

from bs4 import BeautifulSoup


txt = '''<span class="def-price" datasku='....'>
   <i>$</i>
   "999"
   <i>.00<i>
</span>'''

soup = BeautifulSoup(txt, 'html.parser')

print( ''.join(soup.select_one('.def-price').find_all(text=True, recursive=0)).strip() )

Prints:

"999"
user_100359850

Scrapy implements an extension for that as it isn't standard for CSS selectors. So this should work for you:

product_price_sn = product.css('.def-price i::text').extract()

Here is what the docs say:

Per W3C standards, CSS selectors do not support selecting text nodes or attribute values. But selecting these is so essential in a web scraping context that Scrapy (parsel) implements a couple of non-standard pseudo-elements:

to select text nodes, use ::text

to select attribute values, use ::attr(name) where name is the name of the attribute that you want the value of

user_113263190
Related