Using Python to login to Investors Business Daily and get stock data

user_12678716 0
Viewed 223

General Problem

I am trying to gather stock data from Investors Business Daily (IBD) using Python. My goal is to take a stock list and get data for each ticker in the list. The specific page I am interested in is what IBD calls the stock checkup. This page is only viewable for paid subscribers (I am using a free trial).

Specifics

I am trying to use the requests library to login to the session and then use .get to access the stock checkup page. Python version 3.7

import requests

headers = {"User-Agent": "Mozilla/5.0 (Macintosh; Intel Mac OS X 10.14; rv:66.0) Gecko/20100101 Firefox/66.0", "Accept": "text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8", "Accept-Language": "en-US,en;q=0.5", "Accept-Encoding": "gzip, deflate", "DNT": "1", "Connection": "close", "Upgrade-Insecure-Requests": "1"}
LOGIN = 'https://myibd.investors.com/secure/signin.aspx?login'
PROTECTED_PAGE = 'https://research.investors.com/stock-checkup/nasdaq-applied-materials-inc-amat.aspx'
payload = {
    'username': 'blahblah@gmail.com'
    ,'password': 'secretpw'
}

with requests.session() as s:

    s.post(LOGIN, data=payload, headers = headers)
    response = s.get(PROTECTED_PAGE, headers = headers)
    print(response.text)

From other posts I've learned to look for the login form in the html to find the specific tag names for the username, password and other inputs. This is how I found the information for the payload above. I believe there are also hidden inputs that are making this more difficult than I hoped for (eg. __VIEWSTATE). The response text that I receive back indicates that I have not logged in and there is not specific information on the stock of interest (AMAT).

Is there a better way to do this? I have also tried using Selenium but there were issues with that as well. Is the IBD website really that tricky to login to or am I missing something? I've spent countless hours on this so any help is much appreciated!

python-requests html python
1 Answers

For starters, you use incorrect endpoint for authentication.

If the website does not provide a dedicated developer's API, you are restricted to whatever the webapp itself uses.

LOGIN = 'https://myibd.investors.com/secure/signin.aspx?login' - this here is the page where the user enters their credentials. It itself does not and is not expected to accept POST requests.

Whenever the user enters the creds and presses the SUBMIT button, the webapp makes request to the actual authentication endpoint.

You can view requests issued by you browser in the developer's panel, in the "Network" section. And for IBD the request seems to go to https://login.investors.com/accounts.login with the payload (form data)

loginID: afds@asdf.vsd
password: asdfasdgasdfasdfasdf
sessionExpiration: 31536000
targetEnv: jssdk
include: profile,data,emails,subscriptions,preferences,
includeUserInfo: true
loginMode: standard
lang: en
riskContext: {"yadda": "yadda-yadda"}
APIKey: XXXXXXXXXXXXXXX
source: showScreenSet
sdk: js_latest
authMode: cookie
pageURL: https://myibd.investors.com/register/signin-iframe.aspx?checkauth=true&display=&t=1660650471
sdkBuild: 13318
format: json

Now what are the fields in this payload - your guess is probably better than mine. But you gotta nail down the exact request-response ping-pong and the meaning of all fields before trying to build a program around it. Your code does not even check the return code of the authentication attempt - you don't even know if you've logged in successfully or just sent your credentials somewhere.

user_51539320
Related