import multiple files and combine into one large parquet file in s3

Viewed 246

I have multiple files in s3 in one folder:

apc18840407-20191231-01.csv
apc18840407-20191231-02.csv
apc18840407-20191231-03.csv
...apc18840407-20191231-65.csv

because of the multi-header issue, each file needs a bit cleaning before combining with others. The cleaning code:

tm1 = pd.read_csv('s3:file path')
tm1 = tm1.reset_index()
tm1 = tm1.rename(columns=tm1.iloc[0])
tm1 = tm1[6:]
tm1

I am trying to import them all at once and combine the file into a large parquet file.

import boto3 import pandas as pd import io

s3 = boto3.resource('s3')
bucket = s3.Bucket('bucketname')
prefix_objs = bucket.objects.filter(Prefix="s3://folder path")

prefix_df = []

for obj in prefix_objs:
    key = obj.key
    body = obj.get()['Body'].read()
    temp = pd.read_csv(io.BytesIO(body), encoding='utf8')        
    prefix_df.append(temp)

However, this code does not working, and return an empty list. Is there anyway to fix this?

0 Answers
Related