I'm new to data scraping and, recently, I was trying to scrape data from wunderground.com by selenium library with python. However, I found that, sometimes, the selenium web driver cannot successfully open the webpage, I thought this issue may be somewhat related to the JavaScript the website used but not sure which parts went wrong. Does anyone know how to solve it? Thanks in advance.
Here is the example for correctly showing: example for correctly showing
Here shows the problematic one: example for problematic one
My code is here, which is a very simple selenium calls
import requests
from bs4 import BeautifulSoup
import pandas as pd
import numpy
from selenium import webdriver
from selenium.webdriver.support.ui import Select
from selenium.webdriver import ChromeOptions
from selenium.webdriver import ActionChains
import time
# url for scraping
url = "https://www.wunderground.com/history/daily/us/ca/san-diego/KSAN/date/2021-2-1"
# define properties of selenium webdriver
option = webdriver.ChromeOptions()
option.add_experimental_option('excludeSwitches', ['enable-automation'])
option.add_experimental_option('useAutomationExtension', False)
option.add_experimental_option( "prefs",{'profile.managed_default_content_settings.javascript': 1}) #value 1 enables it , if you set to 2 it disables it
option.add_argument('--disable-gpu')
option.add_argument("--disable-blink-features")
option.add_argument("--disable-blink-features=AutomationControlled")
option.add_argument("--enable-javascript")
driver = webdriver.Chrome(options=option)
driver.get(url)
time.sleep(5) # wait for webpage loading