I want to upload a file to S3 bucket using Streaming API of commons-fileupload library. This code is for parsing the request
FileItemIterator iterStream = upload.getItemIterator(request);
while (iterStream.hasNext()) {
FileItemStream item = iterStream.next();
String name = item.getFieldName();
InputStream stream = item.openStream();
if (!item.isFormField()) {
// Process the InputStream stream (*)
} else {
String formFieldValue = Streams.asString(stream);
}
}
This one is for initializing S3 client and transfer manager
s3Client = AmazonS3ClientBuilder.standard()
.withRegion(Regions.DEFAULT_REGION)
.withCredentials(new ProfileCredentialsProvider())
.build();
transferManager = TransferManagerBuilder.standard()
.withS3Client(s3Client)
.build();
I used a 100MB file to test. At the beginning, my springboot app started with about 95MB ram usage. When using that stream(*) to upload to s3 bucket, using
Upload upload = transferManager.upload(bucketName, key, inputStream, metadata );
is significantly more memory consumption (from 90MB->370MB) than copy stream (*) to OutputStream, then upload a file created from that outputStream (from 90MB-> 100MB)
try (
OutputStream out = new FileOutputStream(fileName);
) {
IOUtils.copy(inputStream, out);
}
PutObjectRequest request = new PutObjectRequest(
existingBucketName, fileName, new File(fileName));
Upload upload = transferManager.upload(request);
I wonder why is that. What happened to the inputStream that make it consume more memory to directly upload? Thank you very much