Logging and decoding repsonse body from golang net/http library

user_2989731 0
Viewed 2549

I'm writing a webhook in Go that parses a JSON payload. I'm attempting to log the raw payload and then decode it immediately after but it fails when I try. If I perform the actions separately, they both work fine independently.

Can someone explain why I can't use ioutil.ReadAll and json.NewDecoder together?

func webhook(w http.ResponseWriter, r *http.Request) {
    body, _ := ioutil.ReadAll(r.Body)
    log.Printf("incoming message - %s", body)

    var p payload
    decoder := json.NewDecoder(r.Body)
    err := decoder.Decode(&p)
    if err != nil {
        // Returns EOF
        log.Printf("invalid payload - %s", err)
    }

    defer r.Body.Close()
}
go
3 Answers

Can someone explain why I can't use ioutil.ReadAll and json.NewDecoder together?

The request body is an io.ReadCloser that reads bytes, more or less, directly from a network connection. The contents of the Body aren't stored in memory by default. That's why after the first time you've read the Body the next time you try to read it you'll get EOF.

So if you need to process the request Body more than once, you yourself will have to store the contents into memory, which is what you are already doing with:

body, _ := ioutil.ReadAll(r.Body)

You can then reuse body as many times as you like, and since you have the Body contents at your disposal as a []byte value, you can use json.Unmarshal instead of json.NewDecoder(...).Decode.

This is unrelated to your question, but please do not ignore the error returned from ioutil.ReadAll.

Also you can drop the defer r.Body.Close() line, because you do not have to close the request body in your server handlers. (emphasis mine)

For server requests the Request Body is always non-nil but will return EOF immediately when no body is present. The Server will close the request body. The ServeHTTP Handler does not need to.

user_9659000

r.Body is meant to be read exactly once. When you use the ioutil.ReadAll function you do read all the data from the body. That's why the decoder which also relies on r.Body in fact gets nothing to decode.

user_64830120

Minor additional point about json.Decoder and json.Unmarshal: at first glance it looks like the only difference between the two is just that the former operates on a stream and the latter on a []byte, but they actually have different semantics.

json.Unmarshal will return an error if the data contains more than one json object. So, for example, it will parse {}, but it will not parse {}{}.

json.Decoder parses one complete object per call to Decode, so if you give it {}{}, it will parse those two objects and then the third call will return io.EOF and it's More method will return false.

In a normal http body, you probably only want a single object, so you'd want to use Unmarshal if you're not worried about loading all the data into memory at once. You can also use Decoder and manually check that there is only one object if you care to do so.

user_991890
Related