I have multiple files in s3 in one folder:
apc18840407-20191231-01.csv
apc18840407-20191231-02.csv
apc18840407-20191231-03.csv
...apc18840407-20191231-65.csv
because of the multi-header issue, each file needs a bit cleaning before combining with others. The cleaning code:
tm1 = pd.read_csv('s3:file path')
tm1 = tm1.reset_index()
tm1 = tm1.rename(columns=tm1.iloc[0])
tm1 = tm1[6:]
tm1
I am trying to import them all at once and combine the file into a large parquet file.
import boto3 import pandas as pd import io
s3 = boto3.resource('s3')
bucket = s3.Bucket('bucketname')
prefix_objs = bucket.objects.filter(Prefix="s3://folder path")
prefix_df = []
for obj in prefix_objs:
key = obj.key
body = obj.get()['Body'].read()
temp = pd.read_csv(io.BytesIO(body), encoding='utf8')
prefix_df.append(temp)
However, this code does not working, and return an empty list. Is there anyway to fix this?