I have to import .csv files from AWS S3 in pandas, process on them and than have upload back to S3 in a .csv format as a one single master file. I am using BOTO to make connection with S3 and giving the exact path of the file to import it to local directory. Now after building the process, i want to hit the S3 folder where all the files are residing and import them to local (or may be not), do the processing on top of them and write them back to a different S3 folder in a different bucket in a different folder.
`from boto.s3.connection import S3Connection
from boto.s3.key import Key
import pandas as pd
def get_data():
conn = S3Connection(configuration['aws_access_key_id'],
configuration['aws_secret_access_key'])
bucket = conn.get_bucket(bucket_name=configuration["S3_SURVEY_BUCKET"])
k = Key(bucket)
k.key = 'landing/survey/2015_04_24_WDYWTC.csv'
k.get_contents_to_filename(configuration["SOURCE_FOLDER"])`
My question is how i can achieve it considering the fact that i want to keep one single file for all the data. Any advice is greatly appreciated.