python jupyter list all s3 files using ls command

Admin 1
Viewed 787

I am new to python and am trying to list all the files under s3 directory. I have done this before in databricks using

%sh ls path

I want to understand if there is any similar command available for python- using jupyter based on aws ec2 instance.

I was able to do this using boto3 but I want to understand if there is a simpler way of doing this**(similiar to %sh ls path on databricks)**

import boto3
s3 = boto3.resource('s3')
my_bucket = s3.Bucket('data')

for my_bucket_object in my_bucket.objects.all():
    print(my_bucket_object)
amazon-s3 python
2 Answers

The other way I can think of is to call bash way of listing via python

import os
os.system("aws s3 ls {} --recursive > file_Structure.txt".format("data"))
user_68496820

You could use this to parse out some helpful information (like file size, last modified date):

import pandas as pd
import subprocess
​
def get_s3_files(bucket):
    files = subprocess.Popen(
        args=[f'aws s3 ls s3://{bucket} --recursive'],
        shell=True,
        stdout=subprocess.PIPE
    )
    df = pd.DataFrame(files.communicate()[0].split('\n'))

    df['file_last_modified'] = df[0].apply(
        lambda x: x[:20]
    )
    df['file_size'] = df[0].apply(
        lambda x: x[20:31]
    ).replace('', 0).astype(int)
​
    df['file_name'] = df[0].apply(
        lambda x: x[31:]
    )

    df = df[['file_name',
             'file_last_modified',
             'file_size',]]
​
    return df.sort_values('file_size')

df = get_s3_files(bucket='YOUR_BUCKET')
user_35339600
Related