Get 'None' from website by using beautifulSoup

user_19431663 0
Viewed 50

I am new joiner and doing self study for crawling. I tried to get the information from Disneyland https://www.hongkongdisneyland.com/book/general-tickets/1day-tickets-j

I tried to crawl the price from the website, but it return "None", the result should be HK$639

url5 = 'https://www.hongkongdisneyland.com/book/general-tickets/1day-tickets-j'
r = requests.get(url5)
sp = BeautifulSoup(r.content, 'html.parser')
door = sp.find('div', class_='container')
price = door.find('p', class_='price')
print(price)

My concept of beautifulSoup: It parse the website to be html and I can use find/find_all to find the information by using 'div', 'p', and by its class. Please correct me if it is wrong, thank you.

web-crawler beautifulsoup python
1 Answers

The page is loaded by JavaScript.So to pull out the desired data, you can use selenium with bs4

from bs4 import BeautifulSoup
from selenium import webdriver
from selenium.webdriver.chrome.service import Service
import time
from selenium.webdriver.chrome.options import Options

options = Options()
options.add_argument("--no-sandbox")
options.add_argument("start-maximized")
options.add_experimental_option("detach", True)
webdriver_service = Service("./chromedriver") #Your chromedriver path
driver = webdriver.Chrome(service=webdriver_service,options=options)
url = 'https://www.hongkongdisneyland.com/book/general-tickets/1day-tickets-j'
driver.get(url)
#driver.maximize_window()
time.sleep(10)
soup=BeautifulSoup(driver.page_source, 'lxml') 

price = soup.select_one('p:-soup-contains("General Admission:") > strong').text
print(price)

Output:

HK$639
user_128484110
Related