cant obtain all links from bookmark by Scrapy Selector

user3fSK 0
Viewed 25

I export a bookmark from chrome, and want to use Scrapy Selector to obtain all the links, but I can only get parts of the links (250 out of 650)

here is my code

html = r'C:\Users\super\Downloads\Desktop\temp\html\bookmarks_9_13_22.html'
xpath = r'//@href'
with open(html, 'rb') as f:
    source = f.read()
target = Selector(text=source).xpath(xpath).getall()
print(len(target))

Am I doing anything wrong? I am kind of new in scrapy and XPath.

here is the bookmark (html file)

xpath scrapy python
1 Answers

It seems that the unclosed <DT> tags in the file are causing the issue.

Solution: remove the <DT> tags

from scrapy.selector import Selector
html = open(r'C:\Users\super\Downloads\Desktop\temp\html\bookmarks_9_13_22.html', 'rb').read()
root = Selector(text=html.replace(b'<DT>',b''))
final = root.xpath("//@href").getall()
print(len(final))  # <--- 650
userphOc10
Related