I'm trying to GET a URL with a hash mark in it. I know, this is not proper and a hash mark should never be sent to a web server because it's meant to be handled by the browser as a fragment identifier but I specifically want to include a # in my URL. I'm creating a vulnerability scanner and one of the vulnerabilities I'm trying to test for is improper handling of hash marks in the URL.
If I try and send a request like this:
import requests
url = "http://httpbin.org/anything/foo#bar"
req = requests.get(url)
it will only send a GET request to http://httpbin.org/anything/foo and chop off the "#bar" part. I tried creating a custom adapter to change the behavior but it seems like the URL cleaning happens inside the urllib3 or http libraries too. The HTTPAdapter class inside the requests library has a method called request_url that breaks up the URL into sections. It seems like the hash mark is removed in this method but even with a custom adapter that changes this behavior, the hash mark is never sent. I've tried encoding the hash mark as %23 but it doesn't trigger the vulnerability like the actual hash mark would.
The actual request header is constructed inside the putrequest method of the HTTPConnection class of the http library. The method, path, and protocol of the header is constructed with:
request = '%s %s %s' % (method, url, self._http_vsn_str)
# Output: 'GET http://httpbin.org/anything/foo HTTP/1.1'
Is there any way for me to modify this behavior without having to create a bunch of child classes? I know I can just use python's http library and create a child class of the adapter but I'm trying to keep the additional functionality the requests library provides.
Thanks!