This xpath expression response.xpath('//a[@class="playerName"]/@href') in Scrapy shell only returns the latter half of the href I want to scrape.
The website I'm scraping is https://www.premierleague.com/players. My xpath expression is only returning
'/players/63289/Brenden-Aaronson/overview' instead of 'https://www.premierleague.com/players/63289/Brenden-Aaronson/overview'
Why my Xpath expression only returning part of the href and not the whole href?
hareko
0
Viewed 43
1 Answers
There are two types of links on the web: absolute links and relative links.
Web pages usually link to their internal content using relative links. The browser understands that if I'm on example.com and I click on /page-1.html that the link is relative to example.com - example.com/page-1.html
Scrapy sees the links as they are on the website - as relative links - and if you want to convert them to absolute ones you can use urllib.parse.urljoin function:
from urllib.parse import urljoin
relative_url = "/foo.html"
print(urljoin(response.url, relative_url))
# in your case:
relative_url = response.xpath('//a[@class="playerName"]/@href').get()
print(urljoin(response.url, relative_url))
Related