BeautifulSoup Scraping Only Part of URL

user_15913281 0
Viewed 96

I'm trying to scrape a list of URLs from a page into a list to then loop a scraper through. The code below works but it only scrapes part of the URL instead of the whole one. Eg: /racecards/perth/13-05-21 instead of https://www.horseracing.net/racecards/racecards/perth/13-05-21.

How do I get the full URL?

import requests
from bs4 import BeautifulSoup

URL = "https://www.horseracing.net/racecards"
reqs = requests.get(URL)
content = reqs.text
soup = BeautifulSoup(content, "html.parser")
urls = []

for h in soup.findAll("h3"):
    a = h.find("a")
    try:
        if "href" in a.attrs:
            url = a.get("href")
            urls.append(url)
    except:
        pass

for url in urls:
    print(url)
beautifulsoup python
1 Answers

You can use urllib.parse.urljoin which is better than joining them as a string.

from urllib.parse import urljoin

base_url = "https://www.horseracing.net"

# code omitted ...
# [...]

# when you grab the link, you add the base_url then you have the complete link
url = urljoin(base_url, a.get('href'))
# url is now: https://www.horseracing.net/racecards/perth/13-05-21
user_39543790
Related