Get sibling with xpath

user_13072638 0
Viewed 40

I try to get sibling with this code, any idea why it doesn't print it please ?

url = 'http://www.journaldunet.com/management/ville/lille/ville-59350/immobilier'
res = session.get(url)
price = lxml.html.fromstring(res.text).xpath("//text/tspan[text()='Prix médian']/following-sibling::tspan[3]/text()")
print(res.status_code,price)

On this html which is in http://www.journaldunet.com/management/ville/lille/ville-59350/immobilier

<text x="8" style="font-size:14px;color:#333333;fill:#333333;" y="22">
   <tspan style="font-weight:bold">Prix médian</tspan>
   <tspan style="fill:#3f85f2" x="8" dy="17">●</tspan>
   <tspan dx="0"> Lille: </tspan>
   <tspan style="font-weight:bold" dx="0">2 968 euros</tspan>
</text>

Instead of price, it returns this...

200 []

python-requests lxml siblings xpath python
1 Answers

Because, that html snippet is not in res.text output string. (You may check yourself). It is dynamically created on browser by javascript.

To be able to capture a dynamic page content (like this one), you should use Selenium. It basically loads web page into a real browser (like firefox, chrome etc.), let the browser do its rendering work, and let you grab the data from there.

Here is a nice tutorial how to use and scrape web content.

https://towardsdatascience.com/how-to-use-selenium-to-web-scrape-with-example-80f9b23a843a

user_12878840
Related