Selector.xpath().get() returns all elements after xpath target

user_17677203 0
Viewed 73

Here is my html code that I'm pulling from

from scrapy import Selector
import requests
import pandas as pd

html = '''
<html>
<body>
​
<h1>My First Heading</h1>
​
<p>My first paragraph.</p>
​
</body>
</html>
'''

Then I'm using scrapy selector

sel = Selector( text = html )

I then use xpath to select an element but it also returns everything after the targeted element

in:
sel.xpath('/html/body/h1').get()
out:
'<h1>My First Heading</h1>\n\n<p>My first paragraph.</p>\n\n</body>\n</html>\n'

I'm expecting it to return:

'<h1>My First Heading</h1>'
scrapy web-scraping selector xpath python
1 Answers

The xpath you are using is correct. Your expression returns the expected result on my end. Try the alternative below.

>>> from scrapy.selector import Selector
>>> sel = Selector(text=html)
>>> sel.xpath("//h1").get()
'<h1>My First Heading</h1>'
user_81799390
Related