How to scrape data that is available after events of clicks

user_8150121 0
Viewed 180

I can pull down an HTML page, but not sure how to reach text data hidden under a button click, as the data is nowhere in the page source.

from requests import get

URL = 'https://melvyl.on.worldcat.org/oclc/1076548274'
step1 = get(URL)

print(steps.text)
# how do I navigate to `Check Availability`?

I will like to get the data that is displayed interactively when you click Check Availability next to where it says UC Berkeley Libraries. That opens a box that contains the call number I'm looking for (e.g. "DT157.675 .M37 2019").

hidden web-scraping python
1 Answers

When you monitor Network traffic [In your browser go to More Tools > Developer Tools > Network or press Ctrl + Shift + I in for chrome browser, and select Network, and filter XHR], you will see that when you click Check Availability, the browser makes a get request to another URL to get the data.

from requests import get
from bs4 import BeautifulSoup

# Monitor Post Requests
id_ = 5689
URL = f'https://melvyl.on.worldcat.org/ajax/availabilityFulfillment/oclc/1076548274/registryId/{id_}'
params = {'editionclusteroclcnumbers': 1076548274}

response = get(URL, params=params)

soup = BeautifulSoup(response.text, 'html.parser')
class_name = "availability_call_number_cell availability_left_hand_cell"
results = soup.find('td', class_=class_name).get_text(strip=True)

print(results)
#'DT157.675 .M37 2019'

Example enter image description here

Trying different places, it appears the only thing that change is id_. If you know the ids, then we can collect all data with a loop:


# Monitor Post Requests

# Lets get all ids

URL = 'https://melvyl.on.worldcat.org/ajax/availabilityFulfillment/oclc/1076548274'
params = {'editionClusterOclcNumbers': '1076548274%2C1130899029%2C1126209791'}
response = get(URL, params=params)
soup = BeautifulSoup(response.text, 'html.parser')
id_s = [item['id'].split('_')[-2] for item in soup.find_all("button", {"title":"Check Availability"})]

# get data for all ids
data = []
class_name = "availability_call_number_cell availability_left_hand_cell"
for id_ in id_s:
 
    URL= f'https://melvyl.on.worldcat.org/ajax/availabilityFulfillment/oclc/1076548274/registryId/{id_}'
    params = {'editionclusteroclcnumbers': 1076548274}

    response = get(URL, params=params)

    soup = BeautifulSoup(response.text, 'html.parser')
    
    data.append(soup.find('td', class_=class_name).get_text(strip=True))
    
print(data)
user_68582440
Related