Scraping site behind login with BeautifulSoup but no forms are appearing

Sam Hoppen 0
Viewed 22

I am trying to scrape information from a website that requires you to log in, but when I use BeautifulSoup there aren't any forms that appear on the site. Here's the code I'm using:

import mechanize
import http.cookiejar
from bs4 import BeautifulSoup
# Browser
br = mechanize.Browser()

# Cookie Jar
cj = http.cookiejar.LWPCookieJar()
br.set_cookiejar(cj)

# Browser options
br.set_handle_equiv(True)
br.set_handle_gzip(True)
br.set_handle_redirect(True)
br.set_handle_referer(True)
br.set_handle_robots(False)
br.set_handle_refresh(mechanize._http.HTTPRefreshProcessor(), max_time=1)

#br.addheaders = [('User-agent', 'Chrome')]

# The site we will navigate into, handling it's session
br.open('https://underdogfantasy.com/')
br.forms()

I have to click the "Sign In" button on the website for the prompt to appear, but there isn't a unique URL that's created like there is for some sites. What might I be missing?

beautifulsoup python
1 Answers

There could be 2 issues. One, the website has blocked you from accessing the website using a program which you could check by rendering the HTTP page returned from request.text. Or you need javascript to render the page which could be done by using the html_session library. https://pypi.org/project/requests-html/

walker0
Related