I have ran into a interesting problem. I have a python application in which I scrape multiple websites. One of the websites have cloudflare security implemented, but it's not really strict (I can scrape the website without specifying any means of authentication and it returns the data).
While debugging, I proxy all my python http requests through Fiddler in order to inspect them and see what am I missing and what differs from the request made by browser. I did the same thing here with the code specified below:
response = requests.get(url, headers=headers,
proxies={"http": "http://127.0.0.1:8888", "https": "http://127.0.0.1:8888"},
verify=r"certificate.pem")
As soon as I got it to work, I wanted to remove the proxy so I wouldn't have to run fiddler when I'm running my service. The only thing I changed was the snippet above. I changed it to:
response = requests.get(url, headers=headers)
After running my code, I see that it's returning 403 and that my browser is being verified ('Please stand by, while we are checking your browser...').
I don't think that the http request changed, so I'm assuming something is different in non application level. I'm guessing something fails at the transport level, but before snooping around with wireshark, I decided to ask here instead.