How to select all text content inside div using XPath?

Question

I want to select all text inside a div without considering tags inside.


some text here a link here  span here

I need to get the result as

some text here a link here span here

I tried this

response.xpath('//div/text()')

user_290085 · Answer

You're asking for the string-value of that div:

string(/div)

Or, if you wish whitespace to be trimmed from the ends and consolidated internally:

normalize-space(/div)

user_347767 · Answer

Try to string() it with XPath:

response.xpath('string(//div)').extract_first()

user_9814608 · Answer

check the following code for clarification

response.xpath('//div//text()')

and try the following for the required output

" ".join([i.strip() for i in tree.xpath('//div//text()') if i.strip()])