Why does the HTTP Client Force an Accept-Encoding header

Question

Sample Code:

package main

import (
    "fmt"
    "net/http"
    "net/http/httputil"
)

func main() {
    client := &http.Client{
        Transport: &http.Transport{
            DisableCompression: true,
        },
    }
    url := "https://google.com"
    req, err := http.NewRequest(http.MethodGet, url, nil)
    if err != nil {
        return
    }
    //req.Header.Set("Accept-Encoding", "*")
    //req.Header.Del("Accept-Encoding")
    requestDump, err := httputil.DumpRequestOut(req, false)
    if err != nil {
        fmt.Println(err)
    }

    fmt.Println(string(requestDump))
    client.Do(req)
}

Output:

GET / HTTP/1.1
Host: google.com
User-Agent: Go-http-client/1.1
Accept-Encoding: gzip

With only req.Header.Set("Accept-Encoding", "*" uncommented:

GET / HTTP/1.1
Host: google.com
User-Agent: Go-http-client/1.1
Accept-Encoding: *

With only req.Header.Del("Accept-Encoding") uncommented:

GET / HTTP/1.1
Host: google.com
User-Agent: Go-http-client/1.1
Accept-Encoding: gzip

With both lines uncommented:

GET / HTTP/1.1
Host: google.com
User-Agent: Go-http-client/1.1
Accept-Encoding: gzip

Does DisableCompression actually do anything to the HTTP Request itself? According to the godocs:

    // DisableCompression, if true, prevents the Transport from
    // requesting compression with an "Accept-Encoding: gzip"
    // request header when the Request contains no existing
    // Accept-Encoding value. If the Transport requests gzip on
    // its own and gets a gzipped response, it's transparently
    // decoded in the Response.Body. However, if the user
    // explicitly requested gzip it is not automatically
    // uncompressed.

user_9143962 · Answer

As per document:

DumpRequestOut is like DumpRequest but for outgoing client requests. It includes any headers that the standard http.Transport adds, such as User-Agent.

That means it adds "Accept-Encoding: gzip" to the printed wire format.

To test what is actually written to the connection, you need to wrap Transport.Dial or Transport.DialContext to provide connection that logs written data.

If you are using a transport that supports httptrace (which all built-in and "x/http/..." transport implementation supports), you may set up a WroteHeaderField callback to inspect written header fields.

If you just need to inspect the headers, however, you can spawn up a httptest.Server.

Playground link provided by @EmilePels: https://play.golang.org/p/ZPi-_mfDxI8

Why does the HTTP Client Force an Accept-Encoding header

1 Answers