Hang whilst scraping page using requests in Python

Viewed 182

I am trying to extract a webpage off of the internet, but on some pages (e.g. Tesco's website) I get warning and then it just hangs.

Warning is

InsecureRequestWarning: Unverified HTTPS request is being made to host 'www.tesco.com'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/latest/advanced-usage.html#ssl-warnings

I am using requests to make the connection and get the page.

Minimum working code:

import requests

def extract_page(url):
    page = requests.get(url, verify=False)
    print(f'status code : {page.status_code}')

    if page.status_code != 200:
        print(f"URL '{url} returned status code {page.status_code}")
        return

extract_page('https://www.tesco.com/')

Any guidance on what I am missing would be greatly appreciated, especially as some sites such as Wikipedia work just fine and I couldn't see any other pages that addresses it.

0 Answers
Related