urllib.request.urlopen(url) with Authentication

Admin 1
Viewed 76462

I've been playing with beautiful soup and parsing web pages for a few days. I have been using a line of code which has been my saviour in all the scripts that I write. The line of code is : 

r = requests.get('some_url', auth=('my_username', 'my_password')).

BUT ...

I want to do the same thing with (OPEN A URL WITH AUTHENTICATION):

(1) sauce = urllib.request.urlopen(url).read() (1)
(2) soup = bs.BeautifulSoup(sauce,"html.parser") (2)

I'm not able to open a url and read, the webpage which needs authentication. How do I achieve something like this :

  (3) sauce = urllib.request.urlopen(url, auth=(username, password)).read() (3) 
instead of (1)
request beautifulsoup python-3.x url python
4 Answers

With urllib3 :

import urllib3

http = urllib3.PoolManager()
myHeaders = urllib3.util.make_headers(basic_auth='my_username:my_password')
http.request('GET', 'http://example.org', headers=myHeaders)
user_8121020

Use this. This is standard urllib found with Python3 installation. Works great guaranteed. Also, see gist

import urllib.request

url = 'http://192.168.0.1/'

auth_user="username"
auth_passwd="^&%$$%^"

passman = urllib.request.HTTPPasswordMgrWithDefaultRealm()
passman.add_password(None, url, auth_user, auth_passwd)
authhandler = urllib.request.HTTPBasicAuthHandler(passman)
opener = urllib.request.build_opener(authhandler)
urllib.request.install_opener(opener)

res = urllib.request.urlopen(url)
res_body = res.read()
print(res_body.decode('utf-8'))
user_34867200
Related