sending a request with a proxy causes InvalidURL: Failed to parse in Python

Luca Orlandi 0
Viewed 21

Using Python, I wrote a function that using a proxy, sends a simple request and returns the response, this is the code:

def testSingleProxy(ip,port,user,passw):

    url = 'https://stackoverflow.com/' #example

    proxy = str(ip) + ":" + str(port) + ":" + "@" + str(user) + ":" + str(passw)
    http_proxy =  'http://' + proxy
    https_proxy = 'https://' + proxy

    proxies = {
       'http': http_proxy,
       'https': https_proxy,
    }  

    response = requests.get(str(url), proxies=proxies, timeout=10)
    
    return response

the proxy ip, port, username and password are passed as parameters to the function, each time I try to run the function this error is generated:

Traceback (most recent call last):
  File "/opt/homebrew/lib/python3.10/site-packages/requests/adapters.py", line 456, in send
    conn = self.get_connection(request.url, proxies)
  File "/opt/homebrew/lib/python3.10/site-packages/requests/adapters.py", line 345, in get_connection
    proxy = prepend_scheme_if_needed(proxy, "http")
  File "/opt/homebrew/lib/python3.10/site-packages/requests/utils.py", line 988, in prepend_scheme_if_needed
    parsed = parse_url(url)
  File "/opt/homebrew/lib/python3.10/site-packages/urllib3/util/url.py", line 397, in parse_url
    return six.raise_from(LocationParseError(source_url), None)
  File "<string>", line 3, in raise_from
urllib3.exceptions.LocationParseError: Failed to parse: https://***.***.***.***:****:@*****:*****

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/Users/luca/Desktop/stack", line 28, in <module>
    testSingleProxy("***.***.***.***","****","*****","*****")
  File "/Users/luca/Desktop/stack", line 23, in testSingleProxy
    response = requests.get(str(url), proxies=proxies, timeout=10)
  File "/opt/homebrew/lib/python3.10/site-packages/requests/api.py", line 73, in get
    return request("get", url, params=params, **kwargs)
  File "/opt/homebrew/lib/python3.10/site-packages/requests/api.py", line 59, in request
    return session.request(method=method, url=url, **kwargs)
  File "/opt/homebrew/lib/python3.10/site-packages/requests/sessions.py", line 587, in request
    resp = self.send(prep, **send_kwargs)
  File "/opt/homebrew/lib/python3.10/site-packages/requests/sessions.py", line 701, in send
    r = adapter.send(request, **kwargs)
  File "/opt/homebrew/lib/python3.10/site-packages/requests/adapters.py", line 458, in send
    raise InvalidURL(e, request=request)
requests.exceptions.InvalidURL: Failed to parse: https://***.***.***.***:****:@*****:*****

Note: I have already tried to change both proxy and url, I also tested my proxies with another tester and they works, I also tried to remove any invisible characters in the url but it didn't work. (In the code I covered my proxy numbers and letters with *)

How can I resolve ?

python-requests python-3.x proxy python
1 Answers

The cause of the error

Sorry to say it, but the problem, as is often the case, is a trivial typo right there:

proxy = str(ip) + ":" + str(port) + ":" + "@" + str(user) + ":" + str(passw)
                                     ^

You just using wrong template. The colon should not have come before the @ so parser raised error. In the same way, the credentials should be specified first, and after @ symbol the address and port (like in ssh root@1.1.1.1).

Solution

I can offer this with pretty f-string template, which I use all the time:

def testSingleProxy(ip,port,user,passw):
    url = 'https://stackoverflow.com/' #example

    proxies = {
        "https": f"https://{user}:{passw}@{ip}:{port}",
        "http" : f"http://{user}:{passw}@{ip}:{port}",
    }

    response = requests.get(str(url), proxies=proxies, timeout=10)

    return response
Олексій Холостенко0
Related