Getting href links from a website using Python's Beautiful Soup module

user_17442817 0
Viewed 47

I am trying to get the href links from this page, specifically the links to the pages of those respective clubs. My current code is as follows. I have not included the imports. If needed, I just did import requests and from bs4 import BeautifulSoup:

rsoLink = "https://illinois.campuslabs.com/engage/organizations?query=badminton"
page = requests.get(rsoLink)
beautifulPage = BeautifulSoup(page.content, 'html.parser')
for link in beautifulPage.findAll("a"):
  print(link.get('href'))

My output is empty, suggesting that the program did not find the links. When I looked at the HTML structure of the page, the "a" tags seem to be nested deep within the page's structure (they are inside an element which is within another element, which itself is inside an another element). My question is how I would access the links then; do I have to go through all these elements?

web-scraping beautifulsoup python
2 Answers

The data you see on page is loaded with JavaScript from different URL. So beautifulsoup doesn't see it. To load the data you can use next example:

import json
import requests

url = (
    "https://illinois.campuslabs.com/engage/api/discovery/search/organizations"
)

params = {"top": "10", "filter": "", "query": "badminton", "skip": "0"}

data = requests.get(url, params=params).json()

# uncomment to print all data:
# print(json.dumps(data, indent=4))

for v in data["value"]:
    print(
        "{:<50} {}".format(
            v["Name"],
            "https://illinois.campuslabs.com/engage/organization/"
            + v["WebsiteKey"],
        )
    )

Prints:

Badminton For Fun                                  https://illinois.campuslabs.com/engage/organization/badminton4fun
Illini Badminton Intercollegiate Sports Club       https://illinois.campuslabs.com/engage/organization/illinibadmintonintercollegiatesportsclub
user_100359850

If you take a look at the actual HTML returned by requests, you can see that none of the actual page content is loaded, suggesting that it's loaded client-side via Javascript, likely using an HTTP request to fetch the necessary data.

Here, the easiest solution would be to inspect the HTTP requests made by the site and look for an API endpoint that returns the organizations data. By checking the Network tab of Chrome DevTools, you can find this endpoint:

https://illinois.campuslabs.com/engage/api/discovery/search/organizations?top=10&filter=&query=badminton&skip=0

devtools screenshot

Here, you can see the JSON response for all of the organizations that are being loaded into the page by client-side JS. If you take a look at the JSON, you'll notice that a link isn't one of the keys returned, but it's easily constructed using the WebsiteKey key.

Putting all of this together:

import requests
import json

SEARCH_URL = "https://illinois.campuslabs.com/engage/api/discovery/search/organizations"
ORGANIZATION_URL = "https://illinois.campuslabs.com/engage/organization/"

search = "badminton"
resp = requests.get(
    SEARCH_URL, 
    params={"top": 10, "filter": "", "query": search, "skip": 0}
)

organizations = json.loads(resp.text)["value"]

links = [ORGANIZATION_URL + organization["WebsiteKey"] for organization in organizations]

print(links)

Similar strategies can be used to find and use other API endpoints on the site, such as the organization categories.

user_79412510
Related