Webscraping multiple pages when the url remains the same (but given an ajax response)

user_8116055 0
Viewed 612

I am trying to webscrape all of the reviews for a specific book on Goodreads.com.

url= https://www.goodreads.com/book/show/320.One_Hundred_Years_of_Solitude?ac=1&from_search=true

this worked out pretty successfully for the first page using python and Beautiful Soup, but my problem is trying to scrape the subsequent pages of reviews. I am having issues because each new page that is generated has the same url (so I only get the reviews on page 1). When I inspect the html it seems that the new pages are generated via ajax request. 

<a class="previous_page" href="#" onclick="new Ajax.Request('/book/reviews/320.One_Hundred_Years_of_Solitude?authenticity_token=sZXyhbZUmjF0yvXFy3p2w3PllReMI02adUUeA5yOHzvY1ypaIv1z9e70UMgH1mDpx5FHr%2FakQ4rG7Ge5ZoD6zQ%3D%3D&amp;amp;hide_last_page=true&amp;amp;page=1', {asynchronous:true, evalScripts:true, method:'get', parameters:'authenticity_token=' + encodeURIComponent('4sfXlAmAjNZyCOAnywx+OVJZ1rHkR3E065/m/pbsTC6LhQ9LnSllEug2RSoHoGgT5i0ECZ7AfyRYNp9EbOKp2A==')}); return false;">« previous</a>

I am very new to webscraping in general and have no idea how to go about getting the information I need from this. Any points in the right direction would be awesome.

Thanks

web-scraping beautifulsoup html python ajax
1 Answers

If you're going to be "driving" the web page then I would suggest using a webdriver. https://www.seleniumhq.org/projects/webdriver/

A webdriver can open a "headless" browser that you can manipulate using Selenium's API. For example, in this case you would open the browser and navigate to your page by:

from selenium import webdriver
browser = webdriver.Firefox() # open a browser
browser.get("https://www.goodreads.com/book/show/320.One_Hundred_Years_of_Solitude?ac=1&from_search=true") # open your webpage

Now you're browser object is on the page you are beautiful souping. You can use browser.page_source to get the html, and then soup it:

html = browser.page_source
soup = BeautifulSoup(html, 'html.parser')

Then you can do whatever you want with your soup. When you're ready to get the next page of reviews, you can tell your browser to click the button, wait a second for it to load, then get the soup again:

element = browser.find_element_by_id("your_element_id")
element.click()
time.sleep(3) # sleep three seconds so page can load

html = browser.page_source # now this has new reviews on it
soup = BeautifulSoup(html, 'html.parser') # now you have soup again, but with new reviews

You can throw this process in a loop until there are no more "next page" elements showing up.

user_40283020
Related