Out of memory when uploading large files to S3 using AWS SDK for Java 2.x

Viewed 966

I've got a Java 11 Coretto lambda function where I download files from S3, zip them and upload the zip file to another S3 bucket. This works well with AWS SDK 1.X using TransferManager, but I'm trying not to mix V1 and V2 AWS JAVA SDKs in my project to keep it clean and to minimize the dependencies.

When using the V2 S3Client (software.amazon.awssdk.services.s3) and doing a putObject of a large zip file, I get an OutOfMemory error even though I'm providing an InputStream and setting the contentLength, which I thought would be enough to prevent the whole zip file from being loaded into memory before upload could begin.

Here's my code that causes an OutOfMemory error when trying to upload a large zip file (several gigs):

InputStream targetStream = new FileInputStream(zipFile);

PutObjectRequest request = PutObjectRequest.builder()
        .bucket(zipBucket)
        .key(zipFile.getName())
        .contentType("application/zip")
        .contentLength(zipFile.length())
        .expires(expiration.toInstant()).build();

s3Client.putObject(request, RequestBody.fromInputStream(targetStream, zipFile.length()));

What am I doing wrong?

0 Answers
Related