Make (io.Reader).Read wait for all the bytes to be present

Question

I'm trying to implement a real-time uploading proxy which should take the input raw binary data and forward it into chunks of 250MB each to an external server

I'm currently using Gin as the web framework, and go-resty as the HTTP client handling the upload to the external server

What I have at the moment is:

func chunker(rc io.ReadCloser, bytes int64) ([]byte, int64, error) {
    buf := make([]byte, bytes)
    n, err := rc.Read(buf)
    if err != nil {
        return []byte(""), 0, err
    }
    return buf[:n], int64(n), nil
}

func Upload(c *gin.Context) {
    w := c.Writer
    req := c.Request
    reqBody := req.Body
    reqBodySize := int64(req.ContentLength)
    bytesPerChunk := int64(262144000) // 250 MB

    for {
        var noOfBytes int64 = bytesPerChunk
        if reqBodySize < bytesPerChunk {
            noOfBytes = reqBodySize
        }
        noOfBytes -= 1 // (io.ReadCloser).Read always returns an EOF error if we don't subtract one byte of the total (which is the normal & expected behavior of file uploads by the way)
        if noOfBytes <= 0 {
            break
        }

        bytes, size, err := chunker(reqBody, noOfBytes)
        if err != nil {
            fmt.Println("body EOF")
            break
        }
        reqBodySize -= size

        upload, _, err := external.Upload(bytes)
        if err != nil {
            w.WriteString("an error has occured")
            return
        }
    }

    reqBody.Close()
    w.WriteString("upload complete")
}

Now the issue, is that since the whole process is happening in real time, rc.Read(buf) won't wait till the uploader has finished sending their first 250 MB before jumping to the next chunk

After referring to https://pkg.go.dev/io#Reader, I found out that this is the normal behavior of Read:

Read reads up to len(p) bytes into p. It returns the number of bytes read (0 <= n <= len(p)) and any error encountered. Even if Read returns n < len(p), it may use all of p as scratch space during the call. If some data is available but not len(p) bytes, Read conventionally returns what is available instead of waiting for more.

Is there any way to make Read wait for the specified bytes count to be available instead of using the sent-so-far in the io.ReadCloser instance?

userRBLz · Answer

To finalize this question and get it answered (there still can be better approaches, will appreciate whoever can share them with us)

As stated by @JimB in the comments, io.ReadFull is what I should use to wait for the specified count of bytes to be ready before moving to the next bit of code

So the chunker function should be like:

func chunker(rc io.ReadCloser, bytes int64) ([]byte, error) {
    buf := make([]byte, bytes)
    n, err := io.ReadFull(rc, buf)
    return buf[:n], err
}

Make (io.Reader).Read wait for all the bytes to be present

1 Answers