Decode JSON as it is still streaming in via net/http

user_932548 0
Viewed 5827

In the past I've used go to decode JSON from an API endpoint in the manner shown below.

client := &http.Client{}

req, err := http.NewRequest("GET", "https://some/api/endpoint", nil)
res, err := client.Do(req)
defer res.Body.Close()

buf, _ := ioutil.ReadAll(res.Body)

// ... Do some error checking etc ...

err = json.Unmarshal(buf, &response)

I am shortly going to be working on an endpoint that could send me several megabytes of JSON data in the following format.

{
    "somefield": "value",
    "items": [
        { LARGE OBJECT },
        { LARGE OBJECT },
        { LARGE OBJECT },
        { LARGE OBJECT },
        ...
    ]
}

The JSON will at some point contain an array of large, arbitrary length, objects. I want to take each one of these objects and place them, separately, into a message queue. I do not need to decode the objects themselves.

If I used my normal method, this would load the entire response into memory before decoding it.

Is there a good way to split out each of the LARGE OBJECT items as the response is still streaming in and dispatch it off to the queue? I'm doing this to avoid holding as much data in memory.

go streaming http json
2 Answers

If you want to be as productive as possible, you could read key-value pairs from the stream and tokenize it by your self using lexer from mailru/easyjson library:

r := bufio.NewReader(stream)
for err == nil {
    pair, _ := r.ReadBytes(',')
    x := jlexer.Lexer{
        Data: pair,
    }
    fmt.Printf("%q = ", x.String())
    x.WantColon()
    fmt.Printf("%d\n", x.Int())
}

Note that error handling and some additional checks are skipped for the sake of simplicity. Here the full working example: https://play.golang.org/p/kk-7aEotqFd

user_14731400
Related