How to zip multiple S3 Objects in a single zip file and move it to another folder in the same bucket using C#

user_18950090 0
Viewed 89

I'm trying to write a lambda function which would zip all the s3 objects present in Download folder in a single zip file and then move that zip file to BatchDownload folder in the same s3 bucket.

 ListObjectsRequest downloadS3Object = new ListObjectsRequest
                    {
                        BucketName = sample,
                        Prefix = download
                    };
                    ListObjectsResponse downloadResponse = s3Client.ListObjectsAsync(downloadS3Object).Result;
                    List<string> downloadS3ObjectKeys = downloadResponse.S3Objects.Where(x => !string.IsNullOrEmpty(Path.GetFileName(x.Key)))
                                                                                .Select(s3Object => s3Object.Key)
                                                                                .ToList();

                    foreach (string downloadS3ObjectKey in downloadS3ObjectKeys)
                    {
                        ListObjectsRequest checkBatchDownload = new ListObjectsRequest
                        {
                            BucketName = sample,
                            Prefix = batchDownload
                        };
                        ListObjectsResponse s3ObjectResponse = s3Client.ListObjectsAsync(checkBatchDownload).Result;
                        bool IsArchived = false;
                        if (s3ObjectResponse.S3Objects.Count <= 0)
                        {
                            PutObjectRequest createBatchFolder = new PutObjectRequest()
                            {
                                BucketName = sample,
                                Key = batchDownload
                            };
                            s3Client.PutObjectAsync(createBatchFolder);
                        }

In the above code I'm getting all the objects from download folder and then looping through each of the object keys. I don't understand how to zip all the object keys in a single zip file. Is there a better way to do this without getting the object keys separately. Can you please help with the code to zip all the objects of download folder in a zip file and move that file to a new folder.

amazon-web-services amazon-s3 .net c#
1 Answers

I'm not sure why you appear to be calling ListObjects again, as well as just re-uploading the same objects again, but it doesn't seem right.

It seems you want to download all your objects, place them in a zip archive, and re-upload it.

So you need something like the following:

var downloadS3Object = new ListObjectsRequest
{
    BucketName = sample,
    Prefix = download
};
List<string> downloadS3ObjectKeys;
using (var downloadResponse = await s3Client.ListObjectsAsync(downloadS3Object))
{
    downloadS3ObjectKeys = downloadResponse.S3Objects
        .Where(x => !string.IsNullOrEmpty(Path.GetFileName(x.Key)))
        .Select(s3Object => s3Object.Key)
        .ToList();
}

var stream = new MemoryStream();
using (var zip = new ZipArchive(stream, ZipArchiveMode.Update, true))
{
    foreach (string downloadS3ObjectKey in downloadS3ObjectKeys)
    {
        var getObject = new GetObjectRequest
        {
            BucketName = sample,
            Key = downloadS3ObjectKey,
        };
        var entry = zip.CreateEntry(downloadS3ObjectKey);

        using (var zipStream = entry.Open())
        using (var objectResponse = await s3Client.GetObjectAsync(getObject))
        using (var objectStream = objectResponse.ResponseStream)
        {
            await objectStream.CopyToAsync(zip);
        }
    }
}

stream.Position = 0;  // reset the memorystream to the beginning
var createBatchFolder = new PutObjectRequest()
{
    BucketName = sample,
    Key = batchDownload,
    InputStream = stream,
};

using (await s3Client.PutObjectAsync(createBatchFolder))
{  //
}

Note the use of using to dispose things, and also do not use .Result as you may deadlock, instead use await.

user_148689970
Related