Import CSV from AWS S3 instance to Numpy

user_13487533 0
Viewed 143

I've been trying to directly read a csv file from AWS S3 to numpy. I've used:

s3 = boto3.client(service_name = 's3')

def s3_read(filename):
    s3_obj = s3.get_object(Bucket = 'bucket-name', Key = filename)
    body = s3_obj['Body']
    return body.read()

as an attempt to pull the data but I'm running into an issue of formatting from AWS that I don't know how to handle.

When I print out the data that is being returned from that there is a weird header before the data:

b{\n "name":"filename",\n "data":{\n "type":"Buffer,\n "data:[\n 114,\n 97,...]}}

So there's a bunch of \n's and the weird header. Would this have something to do with the way I uploaded the file to AWS or is there something I'm messing up with the reading of the file?

amazon-web-services csv amazon-s3 numpy python
1 Answers

body.read() returns bytes.

import json
j = json.loads(s3_obj['Body'].read().decode('utf-8'))

decode will turn bytes to string, json.loads will parse the string to dictionary.

user_116333550
Related