How to fix InsecureRequestWarning: Unverified HTTPS request is being made to host

user_14715285 0
Viewed 9858

I am trying to download a zip file from url, but I get the below warning

InsecureRequestWarning: Unverified HTTPS request is being made to host 'www.ons.gov.uk'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/latest/advanced-usage.html#ssl-warnings

I read about this a bit and most of the threads revolves around how to disable it(if you know what you are doing) and the general concept as a whole. From what I understood, isn't request library capable of performing certificate validation by default? Why should I make a secured request to a open source file? I am exposing myself to something here, and how can I fix this ?

Here's my code

import pandas as pd
import requests
from requests.auth import HTTPBasicAuth
from zipfile import ZipFile
import io

url = "https://www.ons.gov.uk/file?uri=%2fpeoplepopulationandcommunity%2fhousing%2fdatasets%2fukhousebuildingpermanentdwellingsstartedandcompleted%2fcurrent/ukhousebuilding.zip"


response = requests.get(url, headers={'User-Agent': 'Mozilla/5.0'}, verify=False)
with ZipFile(io.BytesIO(response.content)) as myzip:
    with myzip.open(myzip.namelist()[0]) as hfile:
        df = pd.read_csv(hfile)

print(df)

Apparently this warning only turns up when I use verify=False, shouldn't it actually prevent this warning?

Sorry for asking too many questions, but I am trying to understand what's happening here. Please correct me if I have misunderstood the concept.

python-requests python
2 Answers

From what I understood, isn't request library capable of performing certificate validation by default?

Yes, it does.

Why should I make a secured request to a open source file?

For security reasons. You want to be sure you actually get the file you expect, from the source you request it from without any manipulations to the file.

I am exposing myself to something here, and how can I fix this?

Yes, you are. Remove the verify=False parameter.

Apparently this warning only turns up when I use verify=False, shouldn't it actually prevent this warning?

No, it's exactly what causes the warning. Read it carefully. It warns you about unverified HTTPS requests. The request is unverified because you specified it.

TL;DR

If the requests works without the verify=False parameter, you should not use it. Otherwise you should still not use it and find another solution.

user_113703120

requests does certificate verification by default on it's own. You have explicitly instructed the library not to do so with verify=False, hence the warning.

You should protect the data transfer as much as possible even in times you might consider requested resources as public. Without proper TLS/SSL validation a man-in-the-middle attacker can intercept your connection. Interception might yield various results:

  • attacker might learn about your communication -- breach of confidentiality
  • attacker might change the contents of the fetched data -- breach of integrity.

Depending on the code using the fetched data, the attacker might be able to influence it's execution and in worst case he/she might try to leverage a security vulnerability in the processing code to achieve your computing system compromise (ultimately a remote code execution).

user_83268670
Related