How to upload files to Amazon S3 (official SDK) that are larger than 5 MB (approx)?

user_191693 0
Viewed 28551

I am using the latest version of the official Amazon S3 SDK (1.0.14.1) to create a backup tool. So far everything works correctly if the size of the file I'm uploading is below 5 MB, but when any of the files is above 5 MB the upload fails with the following exception:

System.Net.WebException: The request was aborted: The request was canceled. ---> System.IO.IOException: Cannot close stream until all bytes are written. at System.Net.ConnectStream.CloseInternal(Boolean internalCall, Boolean aborting) --- End of inner exception stack trace --- at Amazon.S3.AmazonS3Client.ProcessRequestError(String actionName, HttpWebRequest request, WebException we, HttpWebResponse errorResponse, String requestAddr, WebHeaderCollection& respHdrs, Type t) at Amazon.S3.AmazonS3Client.Invoke[T](S3Request userRequest) at Amazon.S3.AmazonS3Client.PutObject(PutObjectRequest request) at BackupToolkit.S3Module.UploadFile(String sourceFileName, String destinationFileName) in W:\code\AutoBackupTool\BackupToolkit\S3Module.cs:line 88 at BackupToolkit.S3Module.UploadFiles(String sourceDirectory) in W:\code\AutoBackupTool\BackupToolkit\S3Module.cs:line 108

Note: 5 MB is roughly the boundary of failure, it can be slightly lower or anything higher

I am assuming that the connection is timing out and the stream is being automatically closed before the file upload completes.

I've tried to find a way to set a long timeout (but I can't find the option in either AmazonS3 or AmazonS3Config).

Any ideas on how to increase the time-out (like an application wide setting I can use) or is it unrelated to a timeout issue?

Code:

var s3Client = AWSClientFactory.CreateAmazonS3Client(AwsAccessKey, AwsSecretKey);

var putObjectRequest = new PutObjectRequest {

    BucketName            = Bucket,
    FilePath              = sourceFileName,
    Key                   = destinationFileName,
    MD5Digest             = md5Base64,
    GenerateMD5Digest     = true
};

using (var upload = s3Client.PutObject(putObjectRequest)) {  }
amazon-s3 .net
5 Answers

I've been having similar problems to this and started to use the TransferUtility class to perform multi part uploads.

At the moment this code is working. I did have problems when the timeout was set too low though! 

                var request = new TransferUtilityUploadRequest()
                .WithBucketName(BucketName)
                .WithFilePath(sourceFile.FullName)
                .WithKey(key)
                .WithTimeout(100 * 60 * 60 * 1000)
                .WithPartSize(10 * 1024 * 1024)
                .WithSubscriber((src, e) =>
                {
                    Console.CursorLeft = 0;
                    Console.Write("{0}: {1} of {2}    ", sourceFile.Name, e.TransferredBytes, e.TotalBytes);
                });
            utility.Upload(request);

As I'm typing this, I have a 4GB upload taking place and it's already got further through than ever before!

user_59320
Related