Closing gzip writer in defer causes data loss

user_7045336 0
Viewed 663

I use golang gzip.NewWriter to zip a slice，and defer Close() to close the writer. But when reading from the zipped data, It will return unexpected EOF. The code is:

func main() {

    a := []byte{'a', 'b', 'c', 'd', 'e', 'f'}
    zippedData, err := zipData(a)
    if err != nil {
        panic(err)
    }

    unzippedData, err := unzipData(zippedData)
    if err != nil {
        panic(err)
    }
    fmt.Printf("%v\n", unzippedData)

}

The zip function is: 

func zipData(originData []byte) ([]byte, error) {
    var bf bytes.Buffer
    gw := gzip.NewWriter(&bf)

    defer gw.Close()

    _, err := gw.Write(originData)
    if err != nil {
        return nil, errors.New(fmt.Sprintf("gzip data err: %v", err))
    }

    err = gw.Flush()
    if err != nil {
        return nil, err
    }
    // if I rm 'defer gw.Close()' and call 'gw.Close()' here, it'll be ok

    logs.Debug("before gzip len: %v", len(originData))
    logs.Debug("gzip len: %v", bf.Len())
    return bf.Bytes(), nil
}

Above zip function uses defer gw.Close() to close gw.

The unzip function is:

func unzipData(zippedData []byte) ([]byte, error) {
    dst := make([]byte, len(zippedData))
    copy(dst, zippedData)

    reader, err := gzip.NewReader(bytes.NewBuffer(dst))
    if err != nil {
        return nil, errors.New(fmt.Sprintf("unzip err :%v", err))
    }

    defer reader.Close()

    data, err := ioutil.ReadAll(reader)
    if err != nil {
        return nil, errors.New(fmt.Sprintf("read err :%v", err))
    }
    return data, err
}

Why defer gw.Close() cases unexpected EOF?

go
2 Answers

With defer, you're missing the gzip footer. According to the Close docs:

Close closes the Writer by flushing any unwritten data to the underlying io.Writer and writing the GZIP footer. It does not close the underlying io.Writer.

So even though Flush flushes any buffered data, it doesn't write the footer. With a deferred close, you get the byte array that doesn't contain the footer and return it, and then the footer is written to the output.

Close the writer before returning.

user_119239990

With defer, gw.Close() runs after the bf.Bytes() call in the return statement. To make sure complete content is returned, you should explicitly call gw.Close before trying to read bytes from the buffer.

The simplest fix for your code is to replace the Flush call with a Close call. Flush is for when you are not done writing, but in your case you've already finished compressing, so calling Close should be enough.

user_2252620
Related