I have this code which sends a request to a website to authenticate with username and password:
from requests import Session
from bs4 import BeautifulSoup as bsoup
session = Session()
r1 = session.get(URL_1)
soup = bsoup(r1.text, 'html.parser')
csrf = soup("input", limit=3)[2]["value"]
data = {"username": "for",
"password": "bar",
"csrf_token": csrf}
r2 = session.post(URL_2, data=data)
print(r2.text)
Above, the r2 is the desired webpage that I want to scrape. Keep in mind URL_1 and URL_2 are different, URL_1 is the webpage for inputting login while URL_2 is the actual login endpoint.
Now this should be the same code using aiohttp.ClientSession:
from aiohttp import ClientSession
import asyncio
from bs4 import BeautifulSoup as bsoup
async def main():
session = ClientSession()
async with session.get(URL_1) as r1:
html = await r1.text()
soup = bsoup(html, 'html.parser')
csrf = soup("input", limit=3)[2]["value"]
data = {"username": "foo",
"password": "bar",
"csrf_token": csrf}
async with session.post(URL_2, data=data) as r2:
print(await r2.text())
await session.close()
loop = asyncio.get_event_loop()
loop.run_until_complete(main())
The r2 html page is still the login page (thats probably the website does after unsuccessful login request)
After, I tried using two separate requests methods without a Session object, and still the second request redirected me to the login page. So, my guess is something has to do with cookies, but what is keeping these two simple pieces of code different)