Cannot gzip slices with more than 32768 bytes in Go 1.5 on Mac OS X

Viewed 332

I am trying to compress byte slices in Go using compress/gzip. Whenever I compress slices with lengths longer than 2^15 on my laptop, every byte with an index of 2^15 or greater is set to 0 after decompression. When I run the same code on my research cluster it also breaks.

Calling go version on my laptop prints:

$ go version
go version go1.5 darwin/amd64

Calling go version on the cluster prints:

$ go version
go version go1.3.3 linux/amd64

Below is a demonstrative test file that I wrote. It generates random slices of different lengths, compresses them, then decompresses them. It checks that no calls returns errors and also checks that the compressed and decompressed slices are the same:

package compress

import (
    "bytes"
    "compress/gzip"
    "math/rand"
    "testing"
)

func byteSliceEq(xs, ys []byte) bool {
    if len(xs) != len(ys) { return false }
    for i := range xs {
        if xs[i] != ys[i] { return false }
    }
    return true
}

func TestGzip(t *testing.T) {
    tests := []struct {
        n int
    }{
        { 1<<10 },
        { 1<<15 },
        { 1<<15 + 1 },
        { 1<<20 },

    }

    rand.Seed(0)

    for i := range tests {
        n := tests[i].n

        in, out := make([]byte, n), make([]byte, n)
        buf := &bytes.Buffer{}
        for i := range in { in[i] = byte(rand.Intn(256)) }

        writer := gzip.NewWriter(buf)
        _, err := writer.Write(in)
        if err != nil {
            t.Errorf("%d) n = %d: writer.Write() error: %s",
                i + 1, n, err.Error())
        }
        err = writer.Close()
        if err != nil {
            t.Errorf("%d) n = %d: writer.Close() error: %s",
                i + 1, n, err.Error())
        }

        reader, err := gzip.NewReader(buf)
        if err != nil {
            t.Errorf("%d) n = %d: gzip.NewReader error: %s",
                i + 1, n, err.Error())
        }
        reader.Read(out)
        err = reader.Close()
        if err != nil {
            t.Errorf("%d) n = %d: reader.Close() error: %s",
                i + 1, n, err.Error())
        }

        if !byteSliceEq(in, out) {
            idx := -1
            for i := range in {
                if in[i] != out[i] {
                    idx = i
                    break
                }
            }
            t.Errorf("%d) n = %d: in[%d] = %d, but out[%d] = %d",
                i + 1, n, idx, in[idx], idx, out[idx])
        }
    }
}

When I run this test, I get the following output:

$ go test --run "TestGzip"
--- FAIL: TestGzip (0.12s)
    gzip_test.go:77: 3) n = 32769: in[32768] = 78, but out[32768] = 0
    gzip_test.go:77: 4) n = 1048576: in[32768] = 229, but out[32768] = 0
FAIL
exit status 1

Does anyone know what is going on here? Am I misusing the package in some way? Let me know if I haven't given enough information.

1 Answers
Related