Python selenium - How to open regular chrome which we use for manual browsing using selenium

user_4855590 0
Viewed 464

I am automating data scraping from the website https://www.macquarieinsights.com/. Now the website needs login only the first time. After that, if you open the website its already logged in. But, when I use selenium to visit the website, it asks for login every time I visit it via selenium. How can I use the regular chrome browser where it's already logged in, using selenium? Or is there any other alternative?

selenium-chromedriver web-scraping selenium python
2 Answers

You can login that site to make the cookies saved in your local browser profile and after that you can use that browser profile.
Existing browser profile settings can be loaded from your PC storage as following:

from selenium import webdriver

options = webdriver.ChromeOptions() 
options.add_argument("user-data-dir=C:\\Path") #Path to your chrome profile
driver = webdriver.Chrome(executable_path="C:\\Users\\chromedriver.exe",chrome_options=options)
user_34854340

Try to save cookies made by the website on your login. And then provide that cookies to selenium while starting it...

Like :

  1. Manually login to browser.

  2. Use requests to get the webpage with the previous cookies

import requests
import browser_cookie3
cookies = browser_cookie3.chrome()
response = requests.get(url, cookies=cookies)

Don't forgot to install module using pip3 install browser-cookie3

Admin1
Related