Move a whole folder from local to S3

user_9837259 0
Viewed 3768

I have to move a folder from my local to s3. I would like to know if there is a way to do so.

My folder contains nested sub-folders with files inside (generally .pdf or .doc or docx).

I know I can move a single file from local to s3 by using s3fs (https://s3fs.readthedocs.io/en/latest/api.html):

S3FileSystem.put(filename, path, **kwargs)  Stream data from local filename to file at path

My code looks like

def upload_data(filepath, file_name):
    s3 = s3fs.S3FileSystem()
    s3_path = f"name-of-my-bucket/{file_name}"
    s3.put(filepath, s3_path)

HOWEVER this allows me to upload a single file. I want to send a whole folder.

I can do it recursively, adding each file one by one, but:

1) I think that it would be easier if I could send the folder.

2) It would be harder to mantain the structure of the folder. Meaning that my local file folders/subfolders/myfile.pdf is going to be saved in s3 as mypdf.pdf rather than folders/subfolders/myfile.pdf

amazon-s3 python
3 Answers

I know this is fairly old, and possibly this functionality didn't exist at the time of asking, but using s3fs you can simply just set recursive=True

def upload_data(filepath, file_name):
    s3 = s3fs.S3FileSystem()
    s3_path = f"name-of-my-bucket/{file_name}"
    s3.put(filepath, s3_path, recursive=True)
user_1417890

using the AWS CLI, you can use the cp command with the --recursive flag:

aws s3 cp folders s3://mybucket/ --recursive
user_121449920

The core AWS S3 API has no function that lets you upload more than one file at a time.

Since all of the higher level implementations of this API tend to mirror the main APIs S3 offers. This includes boto3 and s3fs, they don't add such a feature. If you can't use the AWS cli for some reason, you will need to upload one file at a time, and take care to mirror the local file structure to the remote S3 uploads yourself.

If you do implement this yourself, you'll need to decide the rules for changing local filenames to S3 keys, and probably want to look into using multithreading or multiprocessing to upload a few files at once.

user_23786430
Related