What is the difference between uploading a file to S3 using boto3.resource.put_object() and boto3.s3.transfer.upload_file()

user_5131977 0
Viewed 3267

While I was referring to the sample codes to upload a file to S3 I found the following two ways.

Using boto3.resource.put_object():

s3_resource = boto3.resource('s3')
s3_resource.put_object(Bucket = BUCKET,  Key = 'test', Body= b'some data')

Using boto3.s3.transfer.upload_file():

client = boto3.client('s3')
transfer = S3Transfer(client) 
transfer.upload_file('/my_file', BUCKET, 'test')

I could not figure out the difference between the two ways. Are there any advantages of using one over another in any specific use cases. Can anyone please elaborate. Thank you.

boto3 python-3.x amazon-s3 python
2 Answers
  • The put_object method maps directly to the low-level S3 API request. It will attempt to send the entire body in one request. No multipart support boto3 docs
  • The upload_file method is handled by the S3 Transfer Manager, this means that it will automatically handle multipart uploads behind the scenes for you, if necessary.
    • Automatically switching to multipart transfers when a file is over a specific size threshold
    • Uploading/downloading a file in parallel
    • Progress callbacks to monitor transfers
    • Retries. While botocore handles retries for streaming uploads, it is not possible for it to handle retries for streaming downloads. This module handles retries for both cases so you don't need to implement any retry logic yourself.
    • This module has a reasonable set of defaults. It also allows you to configure many aspects of the transfer process including: Multipart threshold size, Max parallel downloads, Socket timeouts, Retry amounts.boto3 docs
user_64640130
user_22653860
Related