How to use selenium in pandas to read a webpage?

Viewed 157

I want to collect the information of a webpage using chromedriver. How do I install it and use it?

3 Answers

You don't install the chromedriver - you download the .exe (from here) and use the path to it in webdriver.Chrome(). This getting started page has a comprehensive guide:

from selenium import webdriver

driver = webdriver.Chrome('/path/to/chromedriver')  # refers to the path where you saved the exe
driver.get('http://www.google.com/');
time.sleep(5) # Let the user actually see something!
search_box = driver.find_element_by_name('q')
search_box.send_keys('ChromeDriver')
search_box.submit()
time.sleep(5) # Let the user actually see something!
driver.quit()

Note: download the .exe that matches with your version of chrome! (In Help > About Google Chrome)

You have to install selenium first if you don't have it already. Then to use selenium:

from selenium.webdriver import Chrome
url="URL of the webpage you want to read"

setting up the driver

webdriver = "path of the chromedriver.exe file saved in your pc"
driver.get(url)

using css selector

y = driver.find_element_by_css_selector('css selector of the data you want to read from the webpage').text
print(y)

As mentioned by @Patha_Mondal, you need to download the driver and select the elements you want to read. However, as your original question asks "How to use selenium in pandas to read a webpage?", I would say instead consider using Scrapy along with Selenium to create a ".csv" file from the Webpage Data.

Read the ".csv" data into pandas using pandas.read_csv() .

The data from the Webpage might not be clean or properly formatted. Using Scrapy to create a dataset out of it would be beneficial for reading it into pandas. Avoid using pandas directly in the same script as Selenium and Scrapy.

Hope it Helped.

Related