How to save webpage data to a variable in Python

Admin 1
Viewed 1629

I've been looking around for a while to find the answer but unfortunately no luck today.

I'm trying to get the contents off this webpage and save it to a variable. (Link to the website)

Any help would be great, if you are confused what I am meaning drop a question in the comments, because personally I'm not sure how to word this well.

Please don't let me down stackoverflow, I know you can do it ;)

web rss python-3.x python
2 Answers

You'll want to install requests. Hopefully you know pip otherwise please read up on it and install it.

pip install requests

then in your code:

import requests

url = "http://rivalregions.com/rss/all"

req = requests.get(url)

if req.status_code in [200]:
    html = req.text
else:
    print 'Could not retrieve: %s, err: %s - status code: %s' % (url, req.text, req.status_code)
    html = None
user_29734740

You don't need to install requests to get this to work in python3. The below code was tested using python 3.6.10

import urllib.request

def print_some_url():
    try:
        with urllib.request.urlopen('http://www.python.org/') as f:
           a_variable = f.read().decode('utf-8')
           print(a_variable)
    except urllib.error.URLError as e:
       print(e.reason)

print_some_url()
user_9083160
Related