Unable to get text from parent and child nodes/tags with Scrapy

user_5833499 0
Viewed 1787

before this is marked as duplicate, I've searched and tried other solutions found on SO, which are:

  1. scrapy css selector: get text of all inner tags
  2. How to get the text from child nodes if it is parents to other node in Scrapy using XPath
  3. scrapy get the entire text including children

The HTML I want to extract from is:

<span class="location">
    Mandarin Oriental Hotel
    <a class="" href="/search-results/Jalan+Pinang%252C+Kuala+Lumpur+City+Centre%252C+50088+Kuala+Lumpur%252C+Wilayah+Persekutuan./?state=Kuala+Lumpur" itemprop="addressRegion" title="Jalan Pinang, Kuala Lumpur City Centre, 50088 Kuala Lumpur, Wilayah Persekutuan.">
    Jalan Pinang, Kuala Lumpur City Centre, 50088 Kuala Lumpur, Wilayah Persekutuan.
    </a>
    ,
    <a class="" href="/search-results/?neighbourhood=Kuala+Lumpur&state=Kuala+Lumpur" title="Kuala Lumpur">
    Kuala Lumpur
    </a>
    ,
    <a class="" href="/search-results/?state=Kuala+Lumpur" title="Kuala Lumpur">
    Kuala Lumpur
    </a>
    <span class="" itemprop="postalCode">
        50088
    </span>
</span>

I want to get all the text in the //span[@class='location'] .

I have tried:

  1. response.xpath("//span[@class='location']//text()").extract_first()
  2. response.css("span.location *::text").extract_first()
  3. response.css("span.location ::text").extract_first()

All of them only return Mandarin Oriental Hotel, not the full address.

EDIT: The text should yield

Mandarin Oriental Hotel Jalan Pinang, Kuala Lumpur City Centre, 50088 Kuala Lumpur, Wilayah Persekutuan., Kuala Lumpur, Kuala Lumpur 50088

scrapy web-scraping xpath python
2 Answers

Try to use below code to get string representation of each span with address:

for entry in response.xpath("//div[@class='entry']"):
    print(entry.xpath("normalize-space(./span[@class='location'])").extract_first())
user_45495540

With response.css("span.location ::text").extract_first() you get only first text, so you can try to call response.css("span.location ::text").extract() and then concatenate it.

Also you can try get whole parent element and remove tags from it:

from w3lib.html import remove_tags

data = response.css('span.location').get()
if not data:
    return
result = remove_tags(data)
user_71068500
Related