Uploading a file from memory to S3 with Boto3

user_2030941 0
Viewed 6015

This question has been asked many times, but my case is ever so slightly different. I'm trying to create a lambda that makes an .html file and uploads it to S3. It works when the file was created on disk, then I can upload it like so:

boto3.client('s3').upload_file('index.html', bucket_name, 'folder/index.html')

So now I have to create the file in memory, for this I first tried StringIO(). However then .upload_file throws an error.

boto3.client('s3').upload_file(temp_file, bucket_name, 'folder/index.html')
ValueError: Filename must be a string`.

So I tried using .upload_fileobj() but then I get the error TypeError: a bytes-like object is required, not 'str'

So I tried using Bytesio() which wants me to convert the str to bytes first, so I did:

temp_file = BytesIO()
temp_file.write(index_top.encode('utf-8'))
print(temp_file.getvalue())
boto3.client('s3').upload_file(temp_file, bucket_name, 'folder/index.html')

But now it just uploads an empty file, despite the .getvalue() clearly showing that it does have content in there.

What am I doing wrong?

boto3 amazon-web-services python-3.x amazon-s3
2 Answers

If you wish to create an object in Amazon S3 from memory, use put_object():

import boto3

s3_client = boto3.client('s3')

html = "<h2>Hello World</h2>"

s3_client.put_object(Body=html, Bucket='my-bucket', Key='foo.html', ContentType='text/html')
user_1747770

But now it just uploads an empty file, despite the .getvalue() clearly >showing that it does have content in there.

When you finish writing to a file buffer, the position stays at the end. When you upload a buffer, it starts from the position it is currently in. Since you're at the end, you get no data. To fix this, you just need to add a seek(0) to reset the buffer back to the beginning after you finish writing to it. Your code would look like this:

temp_file = BytesIO()
temp_file.write(index_top.encode('utf-8'))
temp_file.seek(0)
print(temp_file.getvalue())
boto3.client('s3').upload_file(temp_file, bucket_name, 'folder/index.html')
user_169383940
Related