Log into a website using Selenium, but continue working (while logged in) with requests

user_13775115 0
Viewed 655

I am using Selenium and the Chrome web driver to log into my account on a website, but after the login, I want to use other libraries (such as requests) to interact with the website.

I am using Selenium to attempt to bypass reCAPTCHA v3, but I want to use the requests and beautifulsoup libraries to scrape data in the URL that comes after the login page (The URL that the login page redirects to, after logging in ).

Here is the code I've written for logging in, and a little snippet at the bottom which I plan to use for scraping the website post-login.

import requests
import os
import time
from selenium import webdriver
from selenium.webdriver.chrome.options import Options
from selenium.webdriver.common.action_chains import ActionChains

chrome_options = Options()
chrome_options.add_argument("--headless")
driver = webdriver.Chrome("chromedriver", options=chrome_options)
action = ActionChains(driver)

url_1 = "https://ais.usvisa-info.com/en-am/niv/users/sign_in"
url_2 = "https://ais.usvisa-info.com/en-am/niv/account/settings/update_email"
email = "email"
password = 'password'
Headers = {
    "user-agent": "Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/89.0.4389.82 Safari/537.36"
}


def login():

    driver.get(url_1)
    
    driver.find_element_by_id("user_email").send_keys(email)
    driver.find_element_by_id("user_password").send_keys(password)
    
    driver.find_elements_by_class_name("icheckbox")[0].click()
    driver.find_elements_by_name("commit")[0].click()
    time.sleep(1)
    print(driver.current_url)

login()
test = requests.get(url, headers=Headers)
python-requests selenium-webdriver web-scraping python
1 Answers

What logging in is actually doing is modifying your cookies to add a key, which verifies that you are logged in. What we can do with this info is to take the cookie data and reuse it for the Python requests module. Let's start by extracting the cookies from the webdriver like so:

driver_cookies = driver.get_cookies()

Now that you have your cookies, you can inject them into future requests in the cookies parameter, like so:

test = requests.get(url, headers=Headers, cookies=driver_cookies)
user_119857430
Related