Stream only partially read when downloading object from Amazon AWS S3

Viewed 1298

I am trying to simply download an object from my bucket using C# just like we can find in S3 examples, and I can't figure out why the stream won't be entirely copied to my byte array. Only the first 8192 bytes are copied instead of the whole stream.

I have tried with with an Amazon.S3.AmazonS3Client and with an Amazon.S3.Transfer.TransferUtility, but in both cases only the first bytes are actually copied into the buffer.

var stream = await _transferUtility.OpenStreamAsync(BucketName, key);
using (stream)
{
    byte[] content = new byte[stream.Length];
    stream.Read(content, 0, content.Length);
    // Here content should contain all the data from the stream, but only the first 8192 bytes are actually populated.
}

When debugging, I see the stream type is Amazon.Runtime.Internal.Util.Md5Stream, and inside the stream, before calling Read() the property CurrentPosition = 0. After the call, CurrentPosition becomes 8192, which seems to indeed indicate only the first 8K of data was read. The total Length of the stream is 104042.

If I make more calls to stream.Read(), I see more data gets read and CurrentPosition increases in value. But CurrentPosition is not a public property, and I cannot access it in my code to make a while() loop (and having to code such loops to read all the data seems a bit clunky).

Why are only the first 8K read in my code? How should I proceed to read the entire stream?

I tried calling stream.Flush(), but it did not fix the problem.

EDIT 1

I have modified my code so it does the following:

var stream = await _transferUtility.OpenStreamAsync(BucketName, key);
using (stream)
{
    byte[] content = new byte[stream.Length];
    var bytesRead = 0;

    while (bytesRead < stream.Length)
        bytesRead += stream.Read(content, bytesRead, content.Length - bytesRead);
}

And it works. But still looks clunky. Is it normal I have to do this?

EDIT 2

Final solution is to create a MemoryStream of the correct size and then call CopyTo(). So no clunky loop anymore and no risk of infinite loop if Read() starts returning 0 before the whole stream has been read:

var stream = await _transferUtility.OpenStreamAsync(BucketName, key);
using (stream)
{
    using (var memoryStream = new MemoryStream((int)stream.Length))
    {
        stream.CopyTo(memoryStream);
        var myBuffer = memoryStream.GetBuffer();
    }
}
1 Answers

stream.Read() returns the number of bytes read. You can then keep track of the total number of bytes read until you have reached the end of the file (content.Length).

You could also just loop until the returned value is 0 meaning error / no more bytes left.

You will need to keep track of the current offset for your content buffer so that you are not overwriting data for each call.

Related