How to proxy https requests using Go, http requests work fine?

user_16770405 0
Viewed 367

I'm going to try my best to explain everything, new to server stuff! Everything is in Go and the main issue is with the ServeHTTP.

Problem

I'm working on forwarding requests received by (http.server).ListenAndServe() to be sent to another proxy server via (http.ReverseProxy).ServeHTTP(http.ResponseWriter, *http.Request). I'm using the python requests to submit a GET request and proxy it through the first server and the second server, to the actual target. I have been able to submit and receive http requests, but I cannot submit https requests.

Error

I constantly receive 502 bad gateway from the python request library, likely a result of http: proxy error: unsupported protocol scheme "" from ServeHTTP() when I use https://httpbin.org or any other https site. Using http results in the html being returned to me.

Response differences

The difference in *http.Response is for http is

&{GET http://httpbin.org HTTP/1.0 1 0 map[] <nil> <nil> 0 [] true http://httpbin.org map[] map[] <nil> map[] [my ip address] http://httpbin.org <nil> <nil> <nil> 0xc000050fc0}

and https is

&{CONNECT //httpbin.org:443 HTTP/1.0 1 0 map[] <nil> <nil> 0 [] true httpbin.org:443 map[] map[] <nil> map[] [my ip address] httpbin.org:443 <nil> <nil> <nil> 0xc0000503c0}

Here is a link to the documentation for *http.Request

What I've previously tried

I've tried modifying ReverseProxy.Director attribute and having the function modify the *http.Request. I've done everything from removing the ":443" and adding "https://" or even "http://". Doing this results in a bad request error. I've also tried changing the CONNECT request to a GET or POST, both of which had strange results including invalid method or bad request.

Any help?

go https proxy http
2 Answers
 ... CONNECT //httpbin.org:443 HTTP/1.0 ...

The client is doing the wrong request. It needs to be CONNECT httpbin.org:443 HTTP/1.0, i.e. just domain and port of the target server. Given that standard Python tools properly handle HTTPS for proxies, the problem is likely caused by some (unknown) self-made code issuing the wrong proxy requests.

user_30810180

I think the issue is that you can't really proxy an https request normally. I assume you're trying to use the server as a proxy to another proxy based on your description, so it looks like you're going to have to Man in the middle it to trick the proxied request to send it along to the actual proxy server.

NOTE this can be kind of a bad idea, make sure that if this is the case all the requests are secure as what'll happen is the intermediate server can read the unencrypted https info.

Unfortunately there's no quick solution I could think of. Instead you have to dive deep into this library and modify the specific MITM request and set up a pool of the further proxy servers that https.go distributes each request to.

Best of luck!

user_170604610
Related