PYTHON 3 - How to web scrape a password protected website?

user_14486159 0
Viewed 3598

I'm trying to access a website in my work, however it's username/password protected. The user/pw pop-up also looks as in the picture.Login image I attach my code to view the website. I can see the HTML code, however with an error "401 Authorization Required". Can you please help?

import requests
from bs4 import BeautifulSoup as bs

r = requests.get("http://10.75.19.101/mfgindex", auth=('root', 'password'))

# Convert to beautiful soup object

soup = bs(r.content, features="html.parser")

# print
print(soup.prettify())
web-scraping python
1 Answers

Generally if site is password-protected you can't obviously bypass the login procedure. That forces you to leverage a RPA process where your code controls the web browser and performs login action leveraging real login and pwd, followed by automated browsing of the pages you need and extraction of the elements you require from HTML using the BeautifulSoup.

For that I suggest to try out Selenium (https://www.selenium.dev/)

A short tutorial is here:

https://medium.com/ymedialabs-innovation/web-scraping-using-beautiful-soup-and-selenium-for-dynamic-page-2f8ad15efe25

I tried it to solve similar task some time ago and it worked good

user_36332500
Related