The Content-MD5 you specified is invalid for multi-part uploads

user_3791924 0
Viewed 1004

I am using the awd-sdk for nodejs and have a working upload.

const checksum = await this.getChecksum(path);
const payload: S3.PutObjectRequest = {
  Bucket: bucket,
  Key: key,
  Body: fs.createReadStream(path),
  ContentMD5: checksum,
};
return this.s3.upload(payload).promise();

This piece of code works great for small files and takes advantage of ContentMD5 which automatically verifies the file integrity.

Content-MD5 The base64-encoded 128-bit MD5 digest of the message (without the headers) according to RFC 1864. This header can be used as a message integrity check to verify that the data is the same data that was originally sent. Although it is optional, we recommend using the Content-MD5 mechanism as an end-to-end integrity check. For more information about REST request authentication, see REST Authentication. https://docs.aws.amazon.com/AmazonS3/latest/API/API_PutObject.html

However it doesn't work for multipart uploads.

The Content-MD5 you specified is invalid for multi-part uploads.

That makes sense because we send the file chunk by chunk but then, I am wondering how am I supposed to use this feature with multipart uploads?

checksum amazon-web-services md5 amazon-s3
1 Answers

I too faced this and did multiple testing. Finally found the answer and verified it in my own way. If you guys know a much better way please let me know here.

Being said that, here's how I solved the issue.

When you create the S3 client you have to create it as below.

const s3 = new AWS.S3({ computeChecksums: true });

Then you can define the s3 upload parameters like below

 var fileData = Buffer.from(fileContentString, 'binary');
 var s3params = {
     Bucket: bucketName,
     Key: folderName + "/" + fileName,
     ContentType: 'binary',
     Body: fileData
 };

Then do the upload as below.

await s3.upload(s3params).promise()
        .then(data => {
            // use this log to verify whether the md5 checksum was verified
            console.log(`File uploadedd successfully ${JSON.stringify(data)}`);
            // handle success upload
        })
        .catch(error => {
            //handle the error
        });

After upload is successful here's how I verified its working.

  • Check the item in the S3. In the document details, Object Overview section check the E-tag. You should see something like this 7e35f58f134d8914604c9fc6c35b2db7-9. This number after the - means how many parts were uploaded. For bigger files there should be a number bigger than 1. In this case it is 9.

  • Check the log in the console. (The log with the comment in above code.) You'll see something like below.

     {
 "Location": "https://bucketname.s3.region.amazonaws.com/folderName/fileName",
 "Bucket": "bucketName",
 "Key": "folderName/fileName",
 "ETag": "\"7e35f58f134d8914604c9fc6c35b2db7-9\""
}

  • If you are debugging you can further test it by printing the s3 upload request.

     var uploadRequest = s3.upload(s3params);
 console.log(`S3 Request ${JSON.stringify(uploadRequest)}`);

    This will print the s3 client configurations. Check whether the 'computeChecksums' is set to true.

I tried to verify with the s3.putObject as well. But when I print the request it didn't show me the md5Checksum config in header as it is intended to be. Also it gave me cyclic JSON stringifying error when I tried to log the whole object in the same way as in the third point for upload. So I printed httpRequest only.

var uploadRequest = s3.putObject(s3params);
console.log(`S3 Request ${JSON.stringify(uploadRequest.httpRequest)}`
//Following gives the JSON stringifying cyclic issue
//console.log(`S3 Request ${JSON.stringify(uploadRequest)}`);

Appreciate if someone can tell how to do this and verify with putObject as well.

user_30358460
Related