Python BeatifulSoup scrape a dynamic container

user_10396263 0
Viewed 63

I am trying to scrape some flashcards from this website, but I am having a few problems. Below a snippet of my code:

# point to the right link and chapter
url_main = r'https://learninglink.oup.com/access/content/neuroscience-sixth-edition-student-resources/neuroscience-6e-chapter-1-flashcards?previousFilter=tag_chapter-'
chapter = '01'
url_main = url_main + chapter

# get source
html = requests.get(url_main).text
bs = BeautifulSoup(html, features="html.parser")

If I inspect the page on Chrome, I can see that the information I am looking for is in class="box1text". So I do:

# get class
text = bs.find(class_ = "box1text" )

However, when I print this 'text' variable I get:

<span aria-live="assertive" class="box1text"></span>

And no mention of the text I am looking for. What am I doing wrong?

Also, I would like to know how to interact with this container and its buttons, but I don't even know where to start from. My ideal output would be a dictionary containing all the keywords and the associated answers (so, front and back of the card for each flashcard), but to do so I need to be able to interact with this container. Any suggestion on how to do this?

Thanks in advance!

web-scraping beautifulsoup html python
1 Answers

I know this doesn't answer your question, but there is actually a better way to go about this.

If you go into devTools sidebar of your browser, and examine the network log, you would see that there is a Http request being sent to get all the flashcard info:screenshot

As you can see, all you need to do is mimic this http request by copying the request header and sending it. Since I don't use python, i would just use cUrl on the windows command prompt. You could do this too right now by right clicking on that exact request when the browser is open and clicking on 'copy as Curl(cmd)' , and pasting whatever you get into the command prompt, and you should be getting the required text that you can easily readenter image description here

Edit: Also, the site in your post doesn't require any additional paramters to be sent in the request, so you should be able to get away with just:

curl "https://learninglink.oup.com/protected/files/content/flashcardCsv/1512079199667-Neuroscience6e-ch01_flashcards.csv"

you can copy and paste this in cmd to verify for yourself

user_163481700
Related