Webscraping content from a webpage not using selenium

user_12467412 0
Viewed 38

I am new to this world of webscraping and its libraries. I wanted to extract certain information from a website, but cannot find the text that I am looking for. This is the website: "https://webgate.ec.europa.eu/rasff-window/screen/notification/486901".

I want to extract information like Product, Product category, reference, subject etc from the website and put it into a dataframe.

The approach that I took was to use the requests and Beatifulsoup library to extract text as suggested in many articles. This is the code that I am using:

from bs4 import BeautifulSoup
import requests

url = 'https://webgate.ec.europa.eu/rasff-window/screen/notification/486901'
res = requests.get(url)
html_page = res.content
soup = BeautifulSoup(html_page, 'html.parser')

text = soup.find_all(text=True)

But when I type text to find out what is inside it, I get something like this:

text

['\n',
'doctype html',
'\n',
'\n',
'\n',
'\n',
'RASFF WINDOW',
'\n',
'\n',
'\n',
'\n',
' FOR OPEN ID IMPLEMENTATION\n\n    <script type="text/javascript" src="/rasff- 
window/assets/jsrsasign-all-min.js"></script>\n    <script type="text/javascript" src="/rasff- 
window/assets/oidc-client.min.js"></script>\n    <script type="text/javascript" src="/rasff-
.
.]

When I use the inspect element I see there is the information I want present in the page but not getting displayed in my text. Also if you check the information is not inside any dynamic javascript, but somehow it is not getting extracted as text.

What should I do in this case, I need to extract information from similar pages(1000's) that's why I was thinking of not using selenium to do the work. Any suggestions or reading is highly appreciated.

web-scraping beautifulsoup python
1 Answers

The data is loaded from external source, so beautifulsoup doesn't see it. You can use requests module to get data from their API:

import json
import requests
import pandas as pd


api_url = "https://webgate.ec.europa.eu/rasff-window/backend/public/notification/view/id/486901/"
data = requests.get(api_url).json()

#uncomment this to print all returned data:
# print(json.dumps(data, indent=4))

# print some data:
print(data["reference"])
print(data["subject"])

# or create a df
#df = pd.json_normalize(data)
#print(df)

Prints:

2021.3617
Salmonella in kipfilet
user_100359850
Related