I'm practicing with the Requests library, and I've run into an issue trying to grab a table from a webpage after first logging in using a session.
I'm trying to grab the "Trading history" tables from pages like this, but in order to see the actual trades in the table, you have to be logged in to the page first. I have login credentials and I've tried using the code below, and it appears to log in fine with a 200 status code, however when I attempt to pull the table from that URL after that, the output of my request still says I need to login to view the table.
The code I'm using looks like this:
import requests
import requests
from bs4 import BeautifulSoup
login_url = "https://www.mql5.com/en/auth_login"
payload = {
'Login': "MY_USERNAME_HERE",
'Password': "MY_PASSWORD_HERE"
}
table_url = "https://www.mql5.com/en/signals/811921?source=Site+Signals+MT5+Table#!tab=history"
# Start the session
session = requests.Session()
# Post the payload to the site to log in
s = session.post(login_url, data=payload)
print(s.status_code)
# Navigate to the next page and scrape the data
html = session.get(table_url)
print(html.status_code)
# Get the soup of the page
soup = BeautifulSoup(html.text, 'html.parser')
# Find the table
table = soup.find("div", {"id": "tab_content_history"})
table = table.find("div", {"class": "tab__content__item"})
print(table)
The output of my prints look like this:
200
200
<div class="tab__content__item">
<div class="need-auth">To see trades in realtime, please <strong><a href="https://www.mql5.com/en/auth_login">log in</a></strong> or <strong><a href="https://www.mql5.com/en/auth_register">register</a></strong></div>
</div>
Login and Password are the name's of the login elements. What else could I be missing in getting the session to login correctly? I don't see any tokens that need to be added to the payload, but I could be wrong as well. What else could I try? If you don't have login to this site it's fine, just looking for other ideas on what to try if you have experience using request sessions. You can review the page inspection of the login form from this page. I recognize web scraping is NOT allowed, I'm just using this site as a public example as I would need to use similar logic for a work project. Thanks!