How to keep logged in on a website using requests library (Python)?

lberj 0
Viewed 32

I have a code that allowed me to web-scrap a website which requires to log in.

I used to log in using Selenium, then get the cookies and put them into a requests.session() like this :

#Install and use the last version of ChromeDriver

driver = webdriver.Chrome(service=Service(ChromeDriverManager().install()))

#Log in to the website

driver.get(url)
driver.find_element_by_name('username').send_keys(e_mail)
driver.find_element_by_name('password').send_keys(mdp)
driver.find_element_by_name('Submit').send_keys('')
maybe_later_css = 'button[class="btn btn-primary btn-lg w-100"]'
driver.find_element_by_css_selector(maybe_later_css).click()

#Get the cookies after being logged in

cookies = driver.get_cookies()

#Put the cookies into a requests.session()

with requests.Session() as s:
for cookie in cookies:
    s.cookies.set(cookie['name'], cookie['value'])

This code worked perfectly for a year but suddenly it didn't anymore. Indeed, when I try a get request on that website, it appears that I'm not logged in (while it worked before).

I tried to add some headers like User-Agent and others, but without success so far.

Here is the website : https://tennis.paris.fr/tennis/jsp/site/Portal.jsp?page=recherche&view=recherche_creneau#!

If anyone has an idea or a solution to fix this, I would be really grateful.

cookies web-scraping python-requests selenium python
1 Answers

I was playing with your code and added some lines, and I got something like this:

#Install and use the last version of ChromeDriver
driver = webdriver.Chrome(service=Service(ChromeDriverManager().install()))

#Log in to the website
driver.maximize_window()
driver.get("https://tennis.paris.fr/tennis/jsp/site/Portal.jsp?page=recherche&view=recherche_creneau#!")
wait = WebDriverWait(driver, 10)
wait.until(EC.element_to_be_clickable((By.CLASS_NAME, "banner-mon-compte__connexion-wrapper")))
driver.find_element(By.CLASS_NAME, "banner-mon-compte__connexion-wrapper").click()
driver.switch_to.window(driver.window_handles[1])
time.sleep(2.0)
driver.find_element(By.NAME, "username").send_keys(e_mail)
driver.find_element(By.NAME, "password").send_keys(mdp)
driver.find_element(By.NAME, "Submit").click()
time.sleep(1.0)
driver.switch_to.window(driver.window_handles[0])

#Get the cookies after being logged in
cookies = driver.get_cookies()
print(cookies)

#Put the cookies into a requests.session()
s = requests.Session()
for cookie in cookies:
    s.cookies.set(cookie['name'], cookie['value'])

print(cookie)

Sorry for amateur automation coding with time.sleep(), wanted to finish this quickly, now you need to switch from window to window when you want to log in to the website.

LukaS0
Related